System and method for automated clustering of clusterable services

Abstract

The system can be configured to automatically deploy clusters of clusterable services. For example, a controller can deploy multiple copies of an application, and these applications can be dependent on each other. The controller can also configure a scheduler that manages these applications (including load balancing). Service templates used by the controller can include clustering rules, which can instruct the controller how to connect these services. Clustering rules can be logic instructions and / or a set of templates that provide for the deployment of services to multiple resources. The binding instructions of a clustering rule define the coordination and interaction of separately reserved physical and / or virtual resources and set dependencies. Clustering rules define the use of information to scale up or scale down resources used by a service.

Description

Technical Field

[0001] Cross - reference to related patent applications and claim of priority: This patent application claims priority to U.S. Provisional Patent Application 63 / 093,691, filed on October 19, 2020, entitled "Systems and Methods for Automatic Clustering of Clustering - Capable Services", the entire disclosure of which is incorporated herein by reference.

[0002] Introduction: As the amount of data generated and consumed by computer users, especially enterprises, continues to increase, there is a technical need for the widespread deployment of high - performance computing (HPC) systems. The attractions of HPC include not only the reduction of computational convergence by parallelization, but also access to huge data memory bandwidths, the ability to schedule compute hardware such as CPUs and graphics processing units (GPUs) for different tasks, integration with artificial intelligence (AI) / machine learning (ML) components, and efficient management of compute hardware resources. Furthermore, the integration of computer - engineering - support systems / electronic design automation (EDA) and AI using HPC is also advancing, recursively feeding the vast amount of data generated by simulation into AI models to analyze and identify optical networks, logic deployment, and process regulation. This integration of EDA and AI / ML accelerates product development and improves quality, but requires a runtime environment that can exhibit optical performance even in a stable, simple, and complex hardware environment.

[0003] Conventionally, HPC has been characterized by low latency, high throughput, large - scale parallel processing, and large - scale distributed systems. For traditional scientific users with computer budgets in the millions of dollars, the cost of software development by information technology (IT) and experts may be only a few percent of the cost of computing time, meaning that the system lacks sufficient ease of setup and usability. As a result, traditional HPC is difficult to use and incurs high personnel costs for operation.

[0004] However, widespread adoption of HPC remains a challenge for many companies. This is because migrating from monolithic workstation-based or custom computing-based platforms to HPC platforms is a non-trivial task. In other words, it is technically difficult to ensure that non-expert users with limited IT budgets and average IT management skills can access HPC applications.

[0005] As a solution to these technical challenges, the inventors disclose a technique for automating the deployment of clusters of clusterable services. A system can be said to "cluster" a service if it runs multiple instances of that service, and these multiple instances work together and can pass instructions to each other. For example, consider a system containing 20 servers running a data mining application. Each of these servers needs to interact with the others, and resources are needed to schedule these interactions. This coordination of clustered services can be a difficult technical challenge, especially for systems running services on bare metal (rather than virtualized). Bare-metal deployment of clustered services is advantageous for services running on a customer's coprocessor or GPU. The term "instance" as used here refers to a service deployed on a resource, which includes, but is not limited to, physical, virtual, or container resources. A cluster can have multiple instances belonging to that cluster.

[0006] These technologies can be used as tools for seamlessly scaling HPC applications from desktops to computer systems with massively parallel environments, which may include deployments across GPU clusters and mixed hardware supporting GPUs. In exemplary embodiments, the computer systems described by the inventors in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088, the entirety of which disclosures are incorporated herein by reference, can be extended to include cluster management services and provide a path for commercially viable automated configuration of clusterable applications in HPC.

[0007] Through such exemplary embodiments, cluster-based computer systems can be used to provide High-Performance Computing as a Service (HPCaaS). HPCaaS is a hybrid of cloud computing and HPC, enabling many users to access HPC at an affordable cost and with relatively little computing time. While traditional HPC systems often utilized one application at a time, HPCaaS offers the ability to use clustered services and storage as a resource pool, a web interface for users to submit job requests, and smart scheduling that allows multiple different applications to be scheduled simultaneously on a specific cluster, taking into account different application characteristics to achieve maximum overall productivity.

[0008] These and other features and advantages of exemplary embodiments of the present invention are described in detail below. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram of a system according to an example embodiment. [Figure 2] Figure 2 is a schematic diagram showing an example of a controller for the system in Figure 1. [Figure 3] Figure 3 shows an example of deploying a service instance on a clustered network. [Figure 4] Figure 4 shows an example of deploying a service instance on a clustered network. [Figure 5] Figure 5 shows an example of a process flow for adding compute resources to a cluster. [Figure 6] Figure 6 shows an example of how a cluster tool manages cluster requests. [Figure 7] Figure 7 shows an example of how a controller manages / calculates dependencies to satisfy cluster dependencies. [Figure 8] Figure 8 shows another example of a service instance deployment in a cluster with a smart clustering-capable network and a storage area network (SAN). [Figure 9] Figure 9 shows an example of a clustering rule. [Figure 10] Figure 10 shows an example where two interdependent clustered services are connected by shared storage. [Figure 11] Figure 11 shows an example of deploying a cluster as a service. [Figure 12] Figure 12 shows an example of the processing flow when the system starts up. [Figure 13] Figure 13 shows examples of various ways to invoke an endpoint. [Figure 14] Figure 14 shows examples of various ways to call an endpoint. [Figure 15] Figure 15 shows the cluster deployed by the controller. [Figure 16A] Figure 16A shows an example of growing a cluster by adding computing resources. [Figure 16B] Figure 16B shows an example of growing a cluster by adding computing resources. [Figure 17A]Figure 17A shows an example of a process flow for creating a new cluster. [Figure 17B] Figure 17B shows an example of a process flow for growing a cluster or adding nodes / resources. [Figure 18A] Figure 18A shows an exemplary process flow for various cluster operations. [Figure 18B] Figure 18B shows an exemplary process flow for various cluster operations.

[0010] Detailed description of the embodiment: Figure 1 shows an exemplary computer system 100 that can be used in connection with the implementation of the clustering technique described herein.

[0011] Examples of system components: The user interface (UI) 110 is shown to be coupled to the controller 200 via an application programming interface (API) application 120. The API 120 may, but does not have to, reside on a standalone physical or virtual server. The API 120 may consist of one or more API applications, which may be redundant and / or run in parallel. The API 120 receives requests that constitute system resources, parses the requests, and passes them to the controller 200. The API 120 receives one or more responses from the controller 200, parses the responses, and passes them to the UI (or application) 110. Alternatively or additionally, applications or services may communicate with the API 120.

[0012] The controller 200 can be deployed in one or more processors and one or more memories to perform any of the control operations discussed herein. Instructions for execution by the processor(s) to perform such control operations can reside in a recording medium containing non-transient computer programs, such as processor memory. The controller 200 is coupled to one or more compute resources 300, storage resources 400, and network resources 500. Thus, the system can include a pool of multiple compute resources 300, multiple storage resources 400, and / or multiple network resources 500 that the controller 200 can configure and control within the system 100. Resources 300, 400, and 500 may reside on a single node, but do not have to, as they may reside on multiple nodes within the system 100 (or they may reside on multiple nodes in various combinations). Also, one or more of the resources 300, 400, and 500 may be virtual. Physical devices may constitute one or more of the resource types, including but not limited to compute resources 300, storage resources 400, and network resources 500. As mentioned above, resources 300, 400, and 500 can constitute a pool of such resources, regardless of whether they are in different physical locations or are virtual. Bare metal compute resources may also be used to enable the use of virtual or containerized compute resources.

[0013] In addition to the known definition of a node, as used herein, a node may be any system, device, or resource connected to a network(s), or any other functional unit performing a function in a standalone or network-connected device. A node may also include, but is not limited to, a server, a service / application / multiple services on a physical or virtual host, a virtual server, and / or multiple or single services on a multitenant server, or a service running within a container.

[0014] One or more processors on which the controller 200 is deployed can take the form of one or more physical or virtual controller servers, which can also operate redundantly and / or in parallel. The controller 200 may operate on a physical or virtual host that functions as a compute host. As an example, the controller 200 can be configured to execute, for example, a controller on a host that serves other purposes in order to have access to confidential resources. The controller 200 receives requests from the API 120, analyzes the requests, performs appropriate tasking on other resources, gives instructions, monitors the resources, receives information from the resources, maintains the state and change history of the system, and may communicate with other controllers that may exist in the system 100. The controller 200 may also include the API 120.

[0015] The compute resources 300 defined herein can constitute a single compute node or a resource pool having one or more physical or virtual compute nodes. The compute resources 300 may include one or more physical or virtual machines or container hosts that can host one or more services or execute one or more applications. The compute resources 300 may also be on hardware designed for multiple purposes including, but not limited to, computing, storage, caching, networking, and / or special computing, such hardware including, but not limited to, GPUs, ASICs, coprocessors, CPUs, FPGAs, and other special computing hardware. Such devices may be added with a PCI Express switch or similar device and may be added dynamically in such a way. The compute resources 300 may be composed of one or more hypervisors or container hosts that include multiple different virtual machines that execute services or applications, or may execute container hosts that can be virtual compute resources. Although the focus of the compute resources may be to provide compute capabilities, they may also include data storage and / or networking capabilities.

[0016] The storage resource 400 as defined herein may constitute a storage node or a pool or storage node. The storage resource 400 may consist of any data storage medium, e.g., fast, slow, hybrid, cache and / or RAM. The storage resource 400 may consist of one or more types of networks, machines, devices, nodes, or any combination thereof, which may or may not be directly connected to other storage resources. According to aspects of exemplary embodiments, the storage resource(s) 400 may be bare metal, virtual, or a combination thereof. While the storage resource may focus on providing storage functionality, it may also include computing and / or networking functionality.

[0017] Network resource(s) 500 can consist of a single network resource, multiple network resources, or a pool of network resources. Network resource(s) 500 may consist of physical or virtual devices(s), tools(s), switches, routers, or other interconnections between system resources, or applications for managing the network. Such system resources may be physical or virtual and may include compute, storage, or other network resources. Network resource(s) 500 provides connectivity between the external network and the application network and may host core network services, including but not limited to Domain Name System (DNS or dns), Dynamic Host Configuration Protocol (DHCP), subnet management, Layer 3 routing, Network Address Translation (NAT), and other services. Some of these services may be deployed on compute resource(s) 300, storage resource(s) 400, or network resource(s) 500, either physically or virtual. The network resource 500 may utilize one or more fabrics or protocols, including but not limited to InfiniBand, Ethernet, RoCE (Remote DMA over Converged Ethernet), Fibre Channel, and / or Omnipath, and may include interconnections between multiple fabrics. The network resource 500 may, but is not required to be, a software-defined network (SDN). The controller 200 may be able to directly modify the network resource 500 or configure the topology of computer systems such as IT systems using SDN, virtual local area networks (VLANs), etc. While the focus of the network resource may be on providing networking functionality, it can also configure compute and / or storage functionality.

[0018] As used herein, an application network refers to a network resource 500, or any combination of network resources 500, for connecting or combining applications, resources, services, and / or other networks, or for connecting users and / or clients to applications, resources, and / or services. An application network can constitute a network used by a server to communicate with other application servers (physical or virtual) and with clients. An application network can communicate with machines or networks outside of system 100. For example, an application network can connect a web frontend to a database. Users can connect to the web application through the internet or another network, which may or may not be managed by controller 200.

[0019] According to an exemplary embodiment, compute, storage, and network resources 300, 400, and 500 can be automatically added, removed, set up, allocated, reassigned, configured, reconfigured, and / or deployed by the controller 200, respectively. According to an exemplary embodiment, additional resources may be added to the resource pool. Examples of techniques for adding, removing, setting up, allocating, reassigning, configuring, reconfiguring, and deploying such resources are described in detail in US Pat. App.Pub. 2019 / 0334909 and WIPO Pub. WO2020 / 252088, referenced and incorporated above.

[0020] Figure 1 shows that a user 105 can access and interact with system 100 via user interface 110. Figure 1 also shows that an application (app) can access, or alternatively access and interact with system 100. For example, a user 105 or an application can send requests to controller 200 via API 120, such requests include, but are not limited to, requests to build an IT system, requests to build individual stacks within an IT system, requests to create a service or application, requests to migrate a service or application, requests to modify a service or application, requests to delete a service or application, requests to replicate a stack to another stack on a different network, and requests to create, add, delete, set up, configure, and / or reconfigure resources or system components. Examples of techniques for performing such requests are described in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088, which are referenced and incorporated above.

[0021] The system 100 in Figure 1 may consist of servers having connections or other communication interfaces to various elements, components, or resources that may be physical, virtual, or any combination thereof. In a modified example, the system 100 shown in Figure 1 may consist of bare-metal servers having connections.

[0022] As detailed in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088 referenced and incorporated above, controller 200 may be configured to power on resources or components, automatically set, configure, and / or control resource startup, add resources, assign resources, manage resources, and / or update available resources. The power-up process may begin by powering on controller 200 so that the order in which devices are started is consistent and does not depend on the user powering on the devices. This process may also include the detection of started resources.

[0023] Figure 2 shows an additional embodiment of the controller 200 within the system 100, which includes controller logic 205, global system rules 210, system states 220, and templates 230.

[0024] Global system rules 210 may declare rules for setting, configuring, starting, allocating, and managing resources, which may include, among other things, computing resources 300, storage resources 400, and network resources 500. Global system rules 210 constitute the minimum requirements for system 100 to be in the correct or desired state. These requirements may consist of tasks that are expected to be completed and an updatable list of expected hardware necessary to build the desired system predictably. The updatable list of expected hardware may allow controller 200 to verify that the necessary resources (e.g., before starting a rule or using a template) are available. Global system rules 210 may consist of a list of operations required for various tasks and corresponding instructions related to the ordering of operations and tasks. For example, rule 210 may specify the order in which components are powered on, the order in which resources, applications and services are started, dependencies, and when different tasks are started, such as loads to configure, start, reload applications, or update hardware. Rule 210 may also include one or more of the following: a list of resource allocations, e.g., a list of resource allocations required for applications and services; a list of available templates; a list of applications to be loaded and how they should be configured; a list of services to be loaded and how they should be configured; a list of application networks and which applications work with which networks; a list of configuration variables specific to different applications and user-specific application variables; a state in which the controller 200 is expected to be able to check the system state 220 to verify that the state is as expected and that the results of each instruction are as expected; and / or a version list including a list of changes to the rule (e.g., snapshots), which may enable tracking changes to the rule and the ability to test or revert to different rules in different circumstances.Controller 200 can be configured to apply global system rules 210 to system 100 based on physical resources, virtual resources, or a combination of physical and virtual resources. Additional information and examples regarding global system rules 210 available to system 100 are provided in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088, referenced and incorporated above.

[0025] Template 230 can constitute a library of templates 230, which may include bare-metal templates and / or service templates. Template 230 can have associations with resources, applications, or services and can function as recipes that define how such resources, applications, or services are integrated into system 100.

[0026] Thus, template 230 may include an established set of information used to create, configure, and / or deploy a resource, or an application or service loaded onto that resource. Such information may include, but is not limited to, the kernel, initrd file, file system or file system image, files, configuration files, configuration file templates, information used to determine the appropriate setup for different hardware and / or compute backends, and / or other available options for configuring resources to run applications and OS images that enable and / or facilitate the creation, booting, or execution of applications.

[0027] Template 230 may contain information that can be used to deploy an application on multiple supported hardware types and / or compute backends, including but not limited to multiple physical server types or components, multiple hypervisors running on multiple hardware types, and container hosts that can be hosted on multiple hardware types.

[0028] Template 230 may derive a boot image for an application or service to run on the computing resource 300. Template 230 and the images derived from Template 230 may be used to deploy applications or services that enable and / or facilitate the creation of applications, and / or to deploy resources for various system functions. Template 230 may have variable parameters for files, file systems, and / or operating system images that can be overridden by configuration options from either default settings or settings provided by the controller 200. Template 230 may have configuration scripts used to configure applications or other resources, and may utilize configuration variables, configuration rules, and / or default rules or variables. These scripts, variables, and / or rules may include specific hardware or other resource-specific parameters, e.g., hypervisor (virtual time), specific rules, scripts, or variables for available memory. Template 230 may have files in the form of binary resources, compilable source code that yields binary resources or hardware or other resource-specific parameters, or a specific set of binary resources or source code that has compilation instructions for specific hardware or other resource-specific parameters, e.g., hypervisor (virtual time), available memory. Template 230 can constitute a set of information independent of what is executed on the resource.

[0029] Template 230 may consist of a base image. The base image may consist of the file system of the base operating system. The base operating system may be read-only. The base image may also consist of basic operating system tools that are independent of what is being run. The base image may include a base directory and operating system tools.

[0030] Template 230 may contain a kernel. The kernel or multiple kernels may contain initrds or multiple kernels configured for different hardware and resource types. An image may be derived from template 230 and loaded or deployed to one or more resources. The captured image may also contain boot files such as the kernel or initrd of the corresponding template 230.

[0031] The image may contain template filesystem information that can be ingested into resources based on template 230. The template filesystem can configure an application or service. For example, the template filesystem can configure a shared filesystem common to all resources or similar resources, for instance, to save storage space where the filesystem is stored or to facilitate the use of read-only files. The template filesystem or image may consist of a set of files common to the services being deployed. The template filesystem may be pre-ingested into the controller or downloaded. The template filesystem may be updated. Because the template filesystem does not require reconstruction, it can enable relatively rapid deployment. Sharing the filesystem with other resources or applications may reduce storage usage because files are not unnecessarily duplicated. It may also facilitate recovery from failures because only files different from the template filesystem need to be restored.

[0032] A template boot file may consist of the kernel and / or initrd or similar filesystem used to assist the boot process. The boot file can start the operating system and set up the template filesystem. The initrd may consist of a small temporary filesystem containing instructions on how to set up template 230 so that it can boot.

[0033] Template 230 may further include template BIOS settings. Template BIOS settings may be used to configure optional settings for running applications on a physical host. If used, the out-of-band management network 260 may be used to start resources or applications. The physical host may start resources or applications using the out-of-band management network 260 or a CD-ROM. The controller 200 may configure application-specific BIOS settings defined in such template 230. The controller 200 can use the out-of-band management network 260 to make direct BIOS changes via APIs specific to particular resources. The settings may be verified through console and image recognition. Thus, the controller 200 can use console functionality to make BIOS changes with a virtual keyboard and mouse. Alternatively, the controller 200 may use a UEFI shell and input directly into the console, using image recognition to verify successful results, confirm that commands are entered correctly, and verify successful configuration changes. If a bootable operating system is available for modifying the BIOS or updating to a specific BIOS version, the controller 200 may run an application that remotely loads the operating system from a disk image or ISO boot, updates the BIOS, and allows the configuration changes to be made in a reliable manner.

[0034] Template 230 may further include a list of template-specific supported resources, or a list of resources required to run a particular application or service.

[0035] The template image, part of the image, or template 230 may be stored in the controller 200, or the controller 200 may move or copy it to the storage resource 400.

[0036] Additional information and examples regarding template 230, which may be used by system 100, are provided in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088, which are referenced and incorporated above.

[0037] System state 220 tracks, maintains, modifies, and updates the state of system 100, including, but not limited to, resources such as compute resources 300, storage resources 400, and network resources 500. System state 220 can track available resources in the form of a database, informing controller logic 205 whether and what resources are available for the implementation of rules 210 and template 230. System state 220 tracks used resources, allowing controller logic 205 to examine efficiency, leverage efficiency, determine if a switch is needed for upgrade or other reasons, and enable a switch to improve efficiency or prioritize. System state 220 may track which applications are running. Controller logic 205 may, according to system state 220, compare expected running applications with actual running applications to determine if any modifications are needed. System state 220 can also track where applications are running. Controller logic 205 can use this information for efficiency assessment, change management, updates, troubleshooting, or audit trails. System state 220 may track network information, such as which networks are on or currently running, or their configuration values ​​and history. System state 220 may also track the history of changes. Furthermore, system state 220 can track which template 230 is used in which deployment, based on global system rules 210 that specify which template 230 is used. History may be used for auditing, alerts, management changes, report building, tracking versions correlated with hardware and applications and configuration, or configuration variables. System state 220 may maintain configuration history for auditing, compliance testing, or troubleshooting purposes.

[0038] Additional information and examples regarding system states 220 that may be used by system 100 are provided in US Pat. App. Pub. 2019 / 0334909 and WIPO Pub. WO 2020 / 252088, which are referenced and incorporated above.

[0039] Controller 200 includes controller logic 205 for managing all information contained in system state 220, template 230, and global system rule 210. Controller logic 205 (which can take the form of an application), global system rule 210, system state 220, and template 230 are managed by controller 200 and may or may not reside in controller 200. Controller logic 205, global system rule 210, system state 220, and template 230 may be physical or virtual. They may, but do not have to be, distributed services, distributed databases, and / or files. API 120 may be included in controller logic 205.

[0040] Controller 200 may run as a standalone machine and / or be composed of one or more controllers. Controller 200 may configure controller services or applications and may run in another machine. The controller machine may start the controller services first to ensure the orderly and / or consistent startup of the entire stack or group of stacks.

[0041] The controller 200 can control one or more stacks having compute, storage, and network resources 300, 400, and 500. Each stack may or may not be controlled by a different subset of rules within the global system rules 210. For example, there may be prototype, product, development, test, parallel, backup, and / or other stacks with different functionalities within the system.

[0042] The controller logic 205 may be configured to read and interpret global system rules 210 to achieve a desired system state. The controller logic 205 may be configured to use templates 230 in accordance with global rules 210 to build system components such as applications or services, allocate, add, or remove resources to achieve a desired state of system 100. The controller logic 205 may read global system rules 210 to create a list of tasks to bring the system into the correct state and issue instructions to satisfy the rules based on available operations. The controller logic 205 may include logic for performing operations (e.g., system startup, resource addition, removal, reconfiguration) and identifying what is possible. The controller logic 205 may check the system state 220 at startup and periodically to determine if hardware is available and, if so, perform tasks. If the required hardware is unavailable, the controller logic 205 presents alternative options using available hardware from global system rules 210, templates 230, and system state 220, and modifies global rules 210 and / or system state 220 accordingly.

[0043] The controller logic 205 can know which variables are needed, what the user needs to input to continue, or what the user needs to make system 100 function. The controller logic 205 can use the list of templates 230 from the global system rule 210 and compare it with the required templates in system state 220 to verify that the required templates are available. The controller logic 205 may also determine from system state 220 whether resources on the template-specific list of supported resources are available. The controller logic 205 can allocate resources, update state 220, and proceed to the next set of tasks to implement the global rule 210. The controller logic 205 can start / run the application on the allocated resources as specified in the global rule 210. Rule 210 can specify how to build the application from template 230. The controller logic 205 can take template 230 and build the application from variables. Template 230 can tell the controller logic 205 which kernels, boot files, file systems, and supported hardware resources are needed. Next, the controller logic 205 can add information about the application deployment to the system state 220. After each instruction, the controller logic 205 can compare the system state 220 with the expected state of the global rule 210 to verify whether the expected operation was completed correctly.

[0044] The controller logic 205 can use versions according to version rules. The system state 220 may have a database related to which rule versions were used in different deployments.

[0045] The controller logic 205 may include efficient logic that defines optimization and efficient ordering. The controller logic 205 may be configured to optimize resources. The controller logic 205 can use information from system states 220, rules 210, and templates 230 related to applications that are running or expected to run to implement efficiency or prioritization regarding resources. The controller logic 205 can use the "used resources" information from system state 220 to determine the need to switch resources for efficiency or for upgrade, reuse, or other reasons.

[0046] The controller 200 may check running applications according to the system state 220 and compare them to the expected running applications of the global rule 210. If an application is not running, it can be started. If an application should not be running, it can be stopped and, if appropriate, resources can be reallocated. The controller logic 205 may include a database of resource (compute, storage network) specifications. The controller logic 205 may include logic for recognizing available resource types on the available system. This may be done using the out-of-band management network 260. The controller logic 205 may be configured to recognize new hardware using the out-of-band management network 260. The controller logic 205 can also retrieve information about change history, rules used, and versions from the system state 220 for auditing, report building, and change management purposes.

[0047] The controller 200 communicates with the stack or resources through one or more of several networks, interconnections, or other connections that the controller 200 can use to direct the compute, storage, and network resources to operate. Such connections may include an out-of-band management connection 260, an in-band management connection 270, a storage area network (SAN) connection 280, and an optional in-band network management connection 290.

[0048] Out-of-band management may be used by controller 200 to discover, configure, and manage components of system 100 via controller 200. An out-of-band management connection 260 may allow controller 200 to discover resources that are plugged in and available but not powered on. Plugged-in resources may be added to system state 220. Out-of-band management may be configured to capture, configure, and monitor boot images belonging to system 100. Out-of-band management can also boot temporary images for operating system diagnostics. Out-of-band management may be used to change BIOS settings and may also use console tools to execute commands on the running operating system. Settings may also be changed by controller 200 using image recognition of video signals from physical or virtual monitor ports on hardware resources such as TA consoles, keyboards, and VGA, DVI, or HDMI ports, and / or using APIs provided by out-of-band management such as Redfish.

[0049] As used herein, out-of-band management may include, but is not limited to, management systems that can connect to resources or nodes independent of the operating system and the main motherboard. An out-of-band management connection may consist of a network or multiple types of direct or indirect connections or interconnections. Examples of out-of-band management connection types include, but are not limited to, IPMI, Redfish, SSH, Telnet, other management tools, Keyboard-Video-Mouse (KVM) or KVM over IP, serial consoles, or USB. Out-of-band management is a tool used over a network that can power on and off nodes or resources, monitor temperature and other system data, make other low-level changes outside the control of the BIOS or operating system, connect to a console and send commands, and control inputs such as keyboards, mice, and monitors. Out-of-band management may be coupled to out-of-band management circuits within physical resources. Out-of-band management may also include a disk image as a disk that can be used to boot an installation medium.

[0050] A management network or in-band management connection 270 may enable the controller 200 to collect information about compute, storage, network, or other resources and to communicate directly with the operating system on which the resources are running. A storage resource, compute resource, or network resource may configure a management interface that interacts with connections 260 and / or 270, thereby communicating with the controller 200, informing the controller 200 what is running and what is available as a resource, and receiving commands from the controller 200. As used herein, an in-band management network consists of a resource and a management network that can communicate directly with the operating system of the resource. Examples of in-band management connections 270 may include, but are not limited to, SSH, Telnet, other management tools, a serial console, or USB.

[0051] Out-of-band management is described herein as a network physically or virtually separated from the in-band management network, but as will be described in more detail herein, these can be combined or operate in conjunction with each other for efficiency purposes. Thus, out-of-band management and in-band management or aspects thereof may communicate through the same port of the controller or be combined in a coupled interconnection. Optionally, one or more connections 260, 270, 280, 290 may be separate from or coupled with others in such networks, and may or may not be part of the same fabric.

[0052] Furthermore, the compute resources, storage resources, and controller may or may not be coupled to the storage network via the SAN connection 280 so that the controller 200 can boot each resource using the storage network. The controller 200 may send a boot image or other template to another storage or other resource so that other resources can boot from storage or other resources. The controller 200 may instruct where to boot from in such a situation. The controller 200 may power on the resource and instruct the resource on where to boot from and how to configure it. The controller 200 instructs the resource on how to boot, what image to use, and where that image is located, if it resides on another resource. The BIOS resources may be pre-configured. The controller 200 may also, or alternatively, configure the BIOS through out-of-band management so that the BIOS boots from the storage area network. The controller 200 may also be configured to boot an operating system from an ISO and allow the resource to copy the data to a local disk. The local disk can then be used for booting. Controller 200 can configure other resources, including other controllers, to enable those resources to be started. Some resources can be configured to provide compute, storage, or networking functions. Furthermore, it is possible for Controller 200 to start a storage resource, after which the storage resource is responsible for supplying boot images for subsequent resources or services. Storage may also be managed on a separate network used for other purposes.

[0053] Optionally, one or more resources may be connected to an in-band management connection 290 on the network. This connection 290 may consist of one or more types of in-band management as described with respect to the in-band management connection 270. The connection 290 may connect the controller 200 to the application network for network utilization or management through the in-band management network.

[0054] Automatic clustering of services: The inventors disclose a number of different techniques that can be implemented by System 100 to automate the deployment of one or more clusters 252 of a clusterable service 250 (see, e.g., Figure 2).

[0055] For example, controller 200 can deploy multiple copies of an application (e.g., n copies of an application, where n can be an integer greater than 1), and these applications can depend on each other. These applications can take the form of services 250. Controller 200 can also configure a scheduler (which may include load balancing 310, shown in Figure 3) that manages these applications (see service instance 250 in Figure 3). As an example, the scheduler can be a cluster manager 302, as shown in Figure 3, which manages cluster 252, manages load balancing, and / or other tasks, and can be a service that schedules those tasks to distribute the processing load. Thus, cluster manager 302 can function as a scheduler that sends out tasks (like SLURM), and other cluster managers 302 may configure various hosts just-in-time. Other services in the environment can depend on cluster 252 of services 250 rather than on just a single service.

[0056] As shown in Figure 4, the service template 430 is used by the controller 200. The service template 430 can be included within template 230. The service template 430 can include clustering rules, which can instruct the controller 200 on how to connect those services. Clustering rules can be logic instructions and / or a set of templates that provide the deployment of a service to multiple resources. The coupling instructions of a clustering rule define the coordination and interaction of separately reserved physical and / or virtual resources and set dependencies. Separate resources can include, but are not limited to, machines, physical, metal, virtual, and / or containers. Clustering rules define the use of information for scaling up or scaling down the resources used by the service. For additional details on an exemplary clustering rule, see Figure 9 below.

[0057] The dotted lines in Figure 4 indicate the connections to each compute resource / service instance within the cluster. These connections can be physical or virtual. Furthermore, if controller 200 must use software-defined networking (SDN) for network resources 500, controller 200 can cluster those services using out-of-band management of the SDN switch (see 260 in Figure 4). For example, the switch's OOB can connect to controller 200 via a serial console, and VLANs can be configured on those ports. As another example, controller 200 can configure OpenSM (InfiniBand subnet manager) on the switch or elsewhere. The SDN can be a network used solely for clustered services 250 to communicate with each other, and such a network configuration can make the system more secure while improving performance.

[0058] Clustering rules can provide load balancing support and can specify a clustering tool 402 (e.g., SLURM (Simple Linux Utility for Resource Management)) that can determine which clustered service is the "master," and the clustering tool(s) can become a dependent service. For example, clustering tool 302 can be defined as a dependency in service template 430. That is, cluster 252 of service 250 may depend on scheduler / clustering tool 302. Also, for example, if service 250 depends on a database service, it may depend on cluster 252 of that service 250. In other exemplary embodiments, service 250 itself may have its own "election" process.

[0059] Figure 9 shows an example of a set of clustering rules 900. These clustering rules 900 contain instructions that cause the controller logic or resource / service instances to manage the cluster. These rules may include, but are not limited to, power-on / off rules and cluster initialization rules. Cluster initialization rules allow the controller logic, cluster manager, and scheduler to initialize cluster resources and configure the resources required for the new cluster.

[0060] These instructions may include hardware-specific instructions that can modify the rules based on supported hardware. These can be done as conditional logic within Rule 900, or Rule 900 can invoke a set of “hardware rules” (these hardware rules specify the supported hardware and what should be done for each type of supported hardware). Hardware types may include information about the base hardware and / or include requirements for expansion cards, including but not limited to network cards, InfiniBand cards, HBAs, disks, GPUs, ASICs, FPGAs, and / or any type of daughter card. Optionally, there are hardware change rules that can change the type of hardware. Often, hardware change rules are complex and not implemented for anything other than simple changes like removing / adding a GPU, but they can be used for any change and can instruct a controller or a daemon with remote power access to the compute resource whose hardware has been changed to restart the resource.

[0061] Service template 430 may indicate that all services deployed from the template must be clustered, and may also indicate that dependent services must be deployed as clusters and that the hardware type of their clustered instances must be the same. Growth / shrink rules in clustering rules (e.g., rules for adding nodes and rules for removing nodes, as shown in Figure 9) may invoke growth / shrink rules on dependent clusters by calling logic within the clustering rules of the dependent services. This could be a storage dependency requiring more disks, where the storage provider is packaged into a service template used as the storage resource provider to the dependent cluster. Clustering rules can also instruct dependent services to function only as dependent services of a particular cluster (e.g., storage, networking, etc., could be in a dedicated pool for a service if there are some software issues with sharing services that would normally be fine to share).

[0062] Cluster initialization rules include programs, logic, and / or instructions for initializing the cluster. Hardware instructions may exist for each required piece of hardware, and the controller 200 may check for any resource requirements. Cluster initialization rules may include calls to endpoints on dependent services. These can send configuration rules to network switches, configure access to storage arrays, reserve data pools, and resolve dependencies required by the cluster (for example, a single instance of a service may only require its own internal storage, while a cluster may require shared storage). Figure 11 shows an example of operations that the system can perform in relation to starting and deploying a cluster.

[0063] Growth rules allow resources to be added to the cluster. These rules generate and provision new resources, which constitute a deployment of a new copy of the clusterable service. The rules can then update specific instructions for all other resources / instances of the service, the cluster manager, and / or the master instance.

[0064] Shrinking rules can invoke cleanup rules to remove instances of resources within a cluster from other resources, preventing dependencies on resources that no longer exist. Cleanup rules can be coupled to growth rules.

[0065] Scaling endpoints and scaling rules can automatically change the cluster configuration at a specific size or suggest user prompts. For example, they might indicate that network bandwidth is saturating the cluster with inter-node communication, or that scaling can be improved after a certain number of nodes have shared storage. Therefore, this rule can mandate dependencies such as storage dependencies for shared file systems.

[0066] Clustering rules allow you to replace endpoints within a service if changes are needed for the cluster. The new endpoint may be hardware-specific. Such endpoint changes are frequent because making changes to the cluster is different from making changes to individual services. Often, but not limited to this case, in the case of a master node, cluster manager, or other cases, the replacement endpoint can be invoked by a dependent service that could be the "cluster manager," and then replaced as an endpoint that makes the same command to all instances of the cluster.

[0067] Clustering rule 900 can also instruct that endpoints run on all resources allocated to the cluster. For example, if there are multiple nodes, controller 200 can remotely log into each node and execute the necessary commands, or it can invoke endpoints that run on each instance / resource depending on the layout (e.g., the endpoint resides on controller 200, and the controller remotely logs in and types the command? Or does it invoke an API endpoint on the machine running the service?).

[0068] Figure 13 shows examples of different ways to invoke an endpoint. For example, controller 200 can invoke a service via an API using in-band management 270. Another example is controller 200 using an endpoint / API to invoke a service (e.g., a script / executable that is part of the service) via OOB 270 (e.g., OOB console). Another example is controller 200 using an endpoint / API to invoke SSH, Telnet, or other remote access on a service via in-band management 270, while otherwise using OOB 260. Figure 14 shows examples of different ways to invoke an endpoint for a cluster.

[0069] Figure 5 shows an exemplary process flow for initializing a new service instance in a cluster. In step 502, the controller 200 provisions service 250 to compute resource 300. In step 504, the controller 200 triggers the creation of service 250 in cluster 252. In step 504, service 250 is started. Next, in step 506, clustering rules are invoked to join service 250 to cluster 252.

[0070] Furthermore, the system can provide clustered services within its own environment, and instead of dependent / dependent services, it can have dependent / dependent clusters (which may also reside within its own environment).

[0071] A clusterable service can include code that runs internally with built-in clustering support. When such a service is packaged as a service template 430, the service template 430 may contain instructions on how to configure the service in a cluster, including network and other infrastructure settings for a clustered deployment, so that the controller automatically configures the service and all instances of the service can communicate properly with each other. Different scenarios can be selected by the user or by rules within the clustering rule 900. For example, given a certain number of nodes, resource usage, and available hardware types (storage, compute, and network—such as InfiniBand or Ethernet), there may be several rules that are automatically generated or suggested to the user. The service template 430 used to deploy a clusterable service to a cluster can be specified by the user or mandated by a service specification file. For example, a service template 430 (e.g., part of a service template 430, which may be in JSON format) may include hardware options along with clustering options, and the service configuration rules may differ based on the hardware being used when they are processed. For example, different base images may exist for different hardware types. Another example is that a different network might be used, or there might be other changes, etc.

[0072] Figure 15 shows a cluster deployed by controller 200. This figure (see Figure 15-1) shows either a clustered service with a deployed service, or a service deployed as a cluster (cluster rules can also be handled as needed, but may not be required for the initial instance). Endpoints can reside on resources / instances or on the controller, and the controller can use remote commands (shown in Figure 14). The initial service is deployed from a service template (see Figure 15-2), and the service image (see Figure 15-3) runs on a resource (typically a compute resource). The compute resource 300 (see Figure 15-5) can be physical, virtual, or a container, and controller 200 can deploy the image onto the resource using ISO, copy it via out-of-band management 260, and copy in-band management, configured files via API, FlexBoot, PXE boot, and / or a combination thereof.

[0073] Cluster rules (see Figure 15, 7) may have shared storage rules (see Figure 15, 8) that can combine compute resources with storage resources or multiple and / or clusters of storage resources. Storage resource 400 may also be deployed as a clustered service, either as a dependency on the current cluster or as a different “resource type”. Figure 15, 15 shows the combination to storage resource 400, which may include, but is not limited to, authentication credentials / public key authentication to the storage resource, the address of the storage resource, connection instructions, and adding InifniBand partitions and / or VLAN tags to one or more connections of compute resources. More generally, all the information necessary to connect to the storage resource, and that the storage resource is properly configured (and that any necessary network resource modifications for combination have been completed).

[0074] Because the cluster uses multiple resources, it shows another resource (see Figure 15, 11) deployed from the service template (see Figure 15, 6) and service image (see Figure 15, 10), and installs configuration rules obtained from both the service template and cluster rules onto the resource (see Figures 15, 7 and 10).

[0075] Cluster rules can verify whether a resource (see 11 in Figure 15) has the appropriate hardware and configure specific hardware-related settings (see "Hardware Instructions" in Figure 9).

[0076] Cluster rules can also include network rules (see Figure 15, item 9). These network rules can be packaged as "additional resource types," and can also be packaged specifically as cluster network rules, as clusters often have their own high-speed networks for rapid interconnection. While these may be general resource types or dependent services deployed as a cluster, most implementations provide dedicated network rules.

[0077] Network rules can bind resources to network resource 500 (see 12 in Figure 15) and can also provision network resources. Network rules (see 9 in Figure 15) can incorporate existing networks and simply include a pointer to a network if a dedicated network does not exist. Network rules can enable ports that connect to compute resources. Network resources may be SANs, but there may also be dedicated SANs or multiple dedicated networks. Network rules may also include load balancing such as DNS round-robin. Multiple network resources and / or networks may be bound to cluster resources. Cluster network rules can directly bind the appropriate network to compute or other resources (see 14 in Figure 15).

[0078] Both network and storage resources require different types of hardware, different storage protocols, network protocols, or network fabrics, and these configuration differences can be derived from hardware rules (see "Hardware Instructions" in Figure 9).

[0079] A cluster can be a cluster manager or depend on a cluster manager (see Figure 15, 18), and service templates can be packaged inside cluster rules. A cluster manager can be a master instance or a separate service. Another instance can be designated as "master" if such designation is required. A cluster manager can configure resources just-in-time, direct how to manage resources within the cluster, and monitor each service running on the cluster. In addition, a cluster manager can act as a scheduler and schedule tasks to various instances within the cluster. Examples of cluster managers include, but are not limited to, a scheduler (such as SLURM), or an instance of a service on the cluster that runs mpirun or other message-passing process initiation tools. Also, if a cluster rule contains logic that a controller can initiate and that controller can schedule those tasks, the controller logic can act as a cluster manager.

[0080] In Figure 16A, unused computing resources are available to grow the cluster (see 20 in Figure 16A). These can be any type of resource, and the diagram is similar to adding storage or network resources. These resources may originally be physically coupled to storage and network, respectively (see 21 and 22 in Figure 16A). The connections can be disabled in a software-defined network, or enabled but not used, and / or the UI can instruct the user to plug in a new cable. Due to the clustering rules, the unused resources are coupled to the cluster.

[0081] The cluster is adding a resource, and Figure 16B is a schematic diagram after the new resource has been added. The controller logic adds the resource using a service template (see 6 in Figure 16B) along with the corresponding cluster rule (see 7 in Figure 16B). The service image (see 24 in Figure 16B) is configured so that the new compute resource 23 becomes part of the cluster and is then joined to all other cluster / dependencies / resources (i.e., compute and storage). All other resources may be updated to take advantage of this new resource (including, but not limited to, references 4, 12, 3, 11, and 18 in Figure 16B). If there is a cluster manager (see 16 / 18 in Figure 16B), it may be updated with information on how to join the new resource (see 23 in Figure 16B) to the cluster.

[0082] Figure 17A shows an example of a process flow for creating a new cluster.

[0083] The clustering rules shown in Figure 9 can have initialization rules. Services may already be deployed or deployed during the initialization step. Initialization rules may have dependencies and / or pointers to other resource types or services to ensure proper operation of the new cluster. An example is shown in label 3 of Figure 16.

[0084] Cluster initialization rules can be executed from the controller, on existing resources, or from the cluster manager service. Initialization rules include instructions on how to build a cluster and combine multiple resources, including but not limited to compute, network, and storage.

[0085] Dependency calculations may exist that can be based on resource allocation 1703. Additional services or instances of clustered services may be deployed 1704. Additional dependencies may exist, in which case other services and / or clustered services may be deployed (e.g., a cluster of object storage for shared storage functionality between cluster compute nodes).

[0086] A service template with cluster rules can have the ability to generate multiple images using the logic and data required within the cluster rules, along with hardware rules for deployment to multiple resource types 1706. In practice, this can be more easily achieved by using one resource type per cluster and including dependencies for additional resource types (for example, a cluster service for object storage can be a dependency) 1705.

[0087] The initialization instructions include logic to combine each resource type and enable all 1707 connections.

[0088] Each instance of a service in the cluster can execute configuration rule 1708, and system state 1709 can be used to recognize the state of each instance on the cluster. Instances of a service in the cluster can use system state to gather information from the controller if inband management 270 is available to collect information from other instances. Alternatively, the cluster manager can push new configurations to each service instance running on the resource.

[0089] Figure 17B shows an example of a process flow for growing a cluster or adding nodes / resources.

[0090] Unused resources are allocated to 1710 (example 20 in Figure 16), and the resources must be physically coupled to system and cluster resources 1711 (examples 21 and 22 in Figure 16). The controller can then process additional / grow rules within cluster rule 1712 (7 in Figure 16). The controller can then derive a service image from the cluster rules, system state, and service templates and / or rules and deploy it to the new resource 1713. Connections to other resource pools, services, or service clusters of the resource can be enabled in 1714 if they were initially disabled.

[0091] Other resources can be joined to the new resource using cluster rules, and logic on each resource in the cluster can be invoked to update the cluster manager, master node, and / or to handle a loop for sending commands to all resources. 1715

[0092] If you are using a load balancer or cluster manager, new resources can be added as part of the list of available resources, along with the logic for connecting to those resources.

[0093] Xyce is an example of a clusterable service. Xyce has built-in OpenMPI support and understands how to use OpenMPI's CUDA support. When packaging Xyce, the service template only needs to know about CUDA-aware OpenMPI configuration and whether to use InfiniBand, Ethernet, or another network fabric. CUDA is the NVIDIA GPU version of C++, and CUDA-enabled OpenMPI sends GPU executable code to any GPU running on a GPU that can be coupled with other hardware (e.g., a server, a CPU coupled with a service instance). Using InfiniBand, for example using NVIDIA NVlink, can be automatically configured to bypass the CPU of the compute node hosting the service. Xyce itself has this support built in, and the service template can be designed to include rules that automatically turn on its clustering capabilities if Xyce is deployed on the appropriate hardware.

[0094] Controller 200 can be provisioned via out-of-band and in-band management (260 / 270) using PXE or IPMI, and can use OOB260 to custom bootloaders and switches to configure multiple applications in a clustered environment, and can combine applications, multiple applications, instances, multiple instances, or combinations thereof. For reference, Controller 200 may be referred to as ASSCM.

[0095] Figure 6 shows an example of a request 600 that may be sent to the cluster tool 402. This request may be from a user or application for a data processing job to be performed by one or more services in cluster 252. The cluster tool 402 can optionally be configured as a dependency of a clusterable service, and the cluster tool 402 can schedule tasks and use message passing tools such as OpenMPI. Clustering rules specified by the service template 430 of the associated cluster 252 may instruct the configuration of the clustering network used to combine clustered services that may be implemented by the controller (see, for example, Figure 9 above). The clustering rules may optionally be used to configure network resources 500 (e.g., switches) via out-of-band management 260.

[0096] Controller 200 can optionally connect to an external network 602 and optionally configure request processing on the cluster tool. This could involve connecting the cluster network, cluster resources, the cluster master instance, and / or the cluster manager to the internet, and / or connecting to another network, either within or outside the system.

[0097] The controller 200's deployment system and dependency management can configure dependencies between services or dependencies between clustered services and dependent or sub-services.

[0098] Figure 7 shows service 702 which depends on another service 704, where service 704 is deployed as cluster 706. [The related cluster 706 consists of two instances of service 704 (these services 704 can be “interdependent” of each other. Interdependence is a simpler way of a cluster where one service has any dependencies on other instances of that service that are currently running. Figure 10 also shows an example where two clustered services are interdependent and coupled with shared storage.

[0099] Figure 8 shows that Controller 200 deploys applications, i.e., clusterable applications that can be arbitrarily deployed on bare metal (e.g., Server 300), via tools including, but not limited to, OOB, IPMI, PXE, Redfish, FlexBoot, custom bootloaders, or combinations thereof. NVlink can be used to bypass the CPU and copy from GPU memory to the memory of another GPU using an InfiniBand connection. Therefore, inter-node communication can be optimized for coprocessors. There may also be a SAN280 or Storage Resource 280 that can provide storage resources between instances of clusterable applications that can be automatically configured on bare metal, or act as a shared storage resource. Network resources may also be configured out-of-band by Controller 200. Switch 800 in Figure 8 (which may consist of multiple switches) can be a switch that connects to compute instances (usually Ethernet) and performs in-band and / or out-of-band management (which may be two switches). SDN Fabric 802 is another switch (e.g., a smart switch) that controller 200 can configure, allowing switch 802 to act as the cluster's high-speed switch, enabling node 300 to communicate with each other very quickly.

[0100] As an example of an embodiment, a system 100 with an automatic clustering function can automatically deploy clusterable applications to bare metal and configure the rest of the system to create a turnkey deployment HPC system environment. As an example, the system boots an ISO, storage resources are connected, and pivot_root is called to move the root file system. Figure 12 shows an example of the process flow for this. In step 1202, the controller 200 provides the ISO image to the virtual CD hardware via the network interface. Alternatively, the virtual CD interface can intelligently request a CD image. In step 1204, the appropriate kernel is loaded from the ISO, and the system boots accordingly (step 1206). In step 1208, the controller 200 provides SAN logon information, where connectivity to the SAN is achieved (step 1210). In step 1212, there may be pivot_root to the new userland.

[0101] As an exemplary embodiment, System 100 includes an out-of-band controller environment designed to enable rapid deployment of network infrastructure and on-demand high-performance applications and services on virtually any hardware. Controller 200 can provide a highly scalable, "clustering-aware" automated deployment system that can reliably scale HPC applications from desktop / workstation environments to massively parallel HPC environments of thousands of nodes, providing VM and container management and / or bare-metal automated deployment. Through awareness of clustered services, applications, and resources, Controller 200 can create, destroy, shrink, and grow clusters 252 in real time. The Controller 200 API, which can be included as part of the clustering rules 900, can include an abstraction layer that provides flexibility to add additional features such as GPU support, cluster security management, and ML interfaces.

[0102] The cluster management API of controller 200 may include an API definition file containing the name, description, argument type, and result type of each API endpoint. Clustering rules 900 may have endpoints for performing "cluster commands". SDKs may exist for these endpoints, and these files may be used to generate API endpoint mappings at runtime. This API generation method makes it relatively easy to develop extensions to the core API when new services or features are added. The server-side implementation of an API endpoint may consist of a mapping between the API endpoint name and a routine that processes arguments, performs the work, and returns an object of the type specified in the API definition.

[0103] Examples of API endpoints that can be included in the API definition file include the following: • Create a new cluster • Destroy the cluster • Grow the cluster • Shrink the cluster • Starting and stopping the cluster • Obtain cluster health • Upgrade the cluster

[0104] Figures 18A and 18B show example process flows for these operations.

[0105] The cluster manager extension of controller 200 can incorporate the ability to orchestrate parallelization across multiple instances of applications and services, as well as the ability to spin up multiple instances of single-user applications. The cluster manager can be responsible for tasks related to managing clustered services and applications, even across interacting clusters, including (1) verifying, tracking, and scheduling changes to the cluster and storing those changes in a persistent database; (2) issuing commands to other managers within controller 200 to recline the resources (such as virtual machines (VMs), storage objects, and networks) required by the cluster; and (3) automatically growing and shrinking the cluster to support those operations.

[0106] In this regard, cluster manager operations may be triggered in response to API calls to the cluster API (such as API calls issued by users) and internally generated automation events. Commands are issued to the domain manager to create new isolation environments for each cluster. These environments may have their own subdomains and subnets. These environments may also have dedicated routers / firewalls (such as routers / firewalls implemented as Linux VMs) to manage traffic inside and outside the cluster. For example, this domain may not be directly managed by the user through the domain API, but rather by the cluster manager. Therefore, in such an example, all administrative operations on the domain may be prohibited unless issued (or permitted) by the cluster manager.

[0107] The domain manager also issues commands to create services that reside within the cluster. These might be N copies of a particular service, or they might include a dedicated scheduler (or control) service that distributes tasks to the nodes in the cluster. This approach allows the executor to work with clustering software that requires a scheduler, or software that can elect its own "leader" to direct the cluster.

[0108] Additionally, you can issue commands to the router / firewall service within the cluster to allow access to the cluster from the domain where the cluster resides.

[0109] Furthermore, commands can be issued to the service manager to deploy and manage each service within the cluster. In an exemplary embodiment, services deployed as part of the cluster cannot be managed directly through the service API. This prevents users from accidentally (or intentionally) modifying parts of the service and causing the cluster to become inconsistent. Instead, services are managed as a group, and changes are applied to all nodes through the cluster manager, ensuring consistency across all cluster members.

[0110] Cluster manager extensions allow you to define service dependencies through managed software deployments and treat a cluster as a single service for the purpose of resolving service dependencies. For example, a cluster's job scheduler needs to access a database service to store job results. The dependency support provided by cluster manager extensions allows clusters to depend on other clusters, which is a desirable characteristic for high-reliability environments.

[0111] The system's service package definition can be updated to include information about the clustering requirements of HPC applications. The new service package definition extensions can be used by the cluster manager to determine how to properly deploy the cluster.

[0112] In another exemplary embodiment, system 100 can incorporate OpenSM automation and management for configuring and securing the InfiniBand fabric in an HPC environment. InfiniBand (IB) is a modern data fabric that enables high-speed (up to 200 Gbps) connectivity between systems, can provide access to high-performance block storage, and also functions as a transport for OpenMPI.

[0113] To achieve this, the local service OpenSM can be adapted to a smart controller that is aware of the state of the HPC system, individual nodes, fabric, components, applications, and parallel computing cluster. Furthermore, the smart controller can configure the interactions between these components to ensure maximum security.

[0114] OpenSM can scan, initialize, and occasionally sweep the IB fabric to respond to changes. OpenSM can first integrate with the controller 200 via an out-of-band connection 260 and then integrate with the network daemon to create the network management daemon (NMD) for system 100. The NMD can then manage automated requests generated by IB configurations and internal system events and services in a way that includes creating, destroying, optimizing, and other methods. The NMD can manage and configure IB hardware on hosts, including route optimization algorithms (which may include minimum hop, DOR routing, and Torus-2 QoS). However, instead of managing each VM or host serially, the NMD can negotiate with each host to optimally configure the IB fabric.

[0115] Support for IB fabrics in clustered systems can be achieved by extending the network API to support the definition and configuration of IB partitions and by adding database tables to track the state of IB and subnet managers. Thus, users will be able to create IB partitions using system 100 and persist them in system 100's internal database. In this regard, the network API specification for system 100 can support the creation of new networks representing IB partitions. This can be achieved by adding support for a new type of network, e.g., "ib-partition". This new ib-partition network type only needs to be supplied with the partition name. Once the network API specification is updated and ib-partition is accepted as a new type of network, a new network plugin can also be adopted. This plugin can be responsible for tracking the state and configuration of each defined IB partition, storing the fabric configuration in a persistent database, and defining the shape of the IB partition data structure consumed by other components within system 100.

[0116] When adding an Ib-partition network to a VM, a GUID can be generated for the Ib interface, which is persistent at VM startup and unique within the deployment. These GUIDs are used by NMD to set the GUID for the SR-IOV virtual function before being passed to QEMU for distribution to the VM.

[0117] A new database table can be added to track the mapping between the ib-partition network and VMs, as well as the GUI used by those VMs. This database table can generate a unique UUID (which can take the form of a 64-bit numeric ID) for every mapping, using specific constraints and built-in database functions. Regardless of which compute host the VM runs on, it will always have the same IB GUID until the device is removed from the VM.

[0118] To consolidate the related functionalities, NMD can get support for configuring the ConnectX VPI card present on the host to enable SR-IOV and setting the GUID of each IB SR-IOV virtual function (VF) to a value controlled by the system. This helps ensure consistency in the fabric topology, as the GUID is created when the ib-partition network is added to the VM. The VM can retain its IB GUID until the network is removed from the VM. To achieve this, SR-IOV is enabled in the Linux kernel image held by the system and it is ensured that Intel VT-x and VT-d or AMD Vi is enabled on the host. This effort allows the use of an in-kernel IB driver rather than the out-of-tree IB driver distributed with Mellanox OFED. The Linux VFIO driver used by QEMU to pass SR-IOV VFs to VMs can also be enabled. NMD can use Linux SysFS to configure SR-IOV on the ConnectX card, configure the VF GUIDs, and combine and uncombine VFs from the Mellanox Driver when the VM needs to access the VFs.

[0119] As part of this initiative, a new internal API extension can be developed in System 100 to allow the Compute Daemon and Controller 200 to request the creation and configuration of SR-IOV VFs for accessing the IB fabric. Four new API functionalities can be adopted for this purpose: • Request an InfiniBand virtual function • Released InfiniBand virtual functions • Get the maximum number of virtual functions • Get the number of virtual functions in use.

[0120] The Request API and Release API can request that all GUIDs necessary to configure or remove a VF be provided, and the VF Utilization API is used to determine if the host can support another VF. If the new VF cannot be configured, the Request API call may report an error. If successful, the request endpoint can return a PCI Bus-Device-Function (BDF) tuple mapped to the virtual function, and the requester can use the new VF.

[0121] Once System 100 is able to manage the SR-IOV VF for IB, it needs to manage OpenSM via Controller 200. This functionality is necessary because Controller 200 can run multiple instances of OpenSM within their own Linux containers, providing redundancy and failover support in case of an OpenSM crash. Each of these containers has its own IB VF that the OpenSM instances can use to configure the fabric, and Controller 200 can be responsible for generating and storing the GUIDs for these interfaces, as it needs to be consistent to ensure fabric stability. Controller 200 is also responsible for generating the necessary OpenSM configuration files and passing them to the containers via a read-only bind mount from the host to the container file system. A separate read / write bind mount can be used to share a per-instance log directory across each container.

[0122] This task can be implemented as a “Worker Plugin” used in a lightweight process management layer that can be integrated into Controller 200. The Worker Plugin defines a process or set of processes that are expected to run on the same host as Controller 200. Currently, this is used to manage local DHCP and HTTP servers used as part of infrastructure orchestration. This new Worker Plugin can launch multiple OpenSM management containers by using existing container runtimes (runc, LXC, rkt, etc.) or by manually creating containers by managing Linux namespaces and control groups (cgroups). While much of the work in this task involves defining the behavior of the containers, generating the OpenSM configuration files is quite straightforward, as the complete specification is published in the Mellanox OFED documentation. Launching the OpenSM management container is divided into eight steps, with steps 2 through 8 repeated for each replica to be deployed: 1. Generate a common OpenSM configuration file. This will include information about partition membership, routing settings, and QoS. 2. Communicate with NMD and create and configure IB SR-IOV VF. 3. Generate instance-specific files: a. The log storage directory where the OpenSM logs accessed by the Controller are stored. b. Instance-specific settings specifying OpenSM Priority for failover support. 4. Use the Container runtime to create a new container containing OpenSM and other OFED components and system packages necessary for it to function correctly. 5. Bind-mount the OpenSM Configuration file to the container as read-only. 6. Bind mount the log directory to the container with read-write permissions. 7. Enable access to the IB VF from within the container. 8. Start OpenSM within the container.

[0123] The base container image can be an Alpine Linux, Gentoo Stage 3 image, or other similarly small and truncated Linux distributions. If OpenSM crashes and the container terminates, one of the replicas can restart or destroy and recreate the container while the fabric takes over management.

[0124] An OpenSM configuration can be generated based on the number of OpenSM replicas defined and the IB partition membership defined by the VM's "ib-partition" network membership, which is implemented as a Network Device attached to the VM. The controller 200 can dictate the priority of each OpenSM instance in the generated per-instance configuration, although the requirement would be to run instances with at least priority 1 and 2.

[0125] QEMU supports PCIe Passthrough through the Linux VFIO driver and specific command-line arguments when QEMU starts. To support this feature, the Compute Daemon can issue an IB SR-IOV VF Request to the NMD on the same host when starting a new VM if the VM has an "ib-partition" network device. If this request fails, the VM will not start; otherwise, the VM will start normally. The next step is to generate the arguments necessary to pass the VF to the VM.

[0126] The Compute Daemon generates QEMU command-line arguments by mapping a list of connected devices to an equivalent set of arguments. Adding a new mapping involves checking the type of device (in this case, a Network Device connected to the "ib-partition" Network), retrieving the settings associated with that device, and constructing the command-line arguments. For PCI Passthrough, this means using the "-device vfio-pci,host=$bdf" argument to tell QEMU which VFs should be passed through to the VM.

[0127] While the present invention has been described above in relation to its exemplary embodiments, various modifications may be made within the scope of the invention. Such modifications to the invention will be apparent from a consideration of the teachings herein.

Claims

1. A system comprising a computer system controller, wherein the controller automatically clusters clusterable services using a plurality of clustering rules defined by a service template, The controller reads the clustering rules defined by the service template, and deploys multiple service instances as a cluster of services corresponding to the clusterable services according to the read clustering rules. The controller deploys a scheduler for managing the service instance. system.

2. The system according to claim 1, wherein the controller selects the service template from a library of templates.

3. The system according to any one of claims 1 to 2, wherein the plurality of service instances are interdependent.

4. The system according to any one of claims 2 to 3, wherein the scheduler manages the service instances by load balancing them.

5. The system according to any one of claims 1 to 4, wherein the clustering rule includes a cluster initiation rule, and the cluster initiation rule identifies the dependencies of the clusters.

6. The system according to any one of claims 1 to 5, wherein the clustering rules include different rules for deploying the cluster based on a plurality of different hardware types on which the cluster is deployed.

7. The system according to claim 6, wherein the hardware type includes a GPU.

8. The system according to any one of claims 1 to 7, wherein the clustering rules include rules for growing the cluster.

9. The system according to claim 8, wherein the rule for growing the cluster includes a pointer to a service image of the cluster.

10. The system according to claim 8, wherein the rules for growing the cluster include networking rules for the cluster.

11. The system according to any one of claims 1 to 10, wherein the clustering rules include rules for reducing the clusters.

12. The system according to any one of claims 1 to 11, wherein the clustering rules include rules that specify the resource requirements of the cluster.

13. The system according to any one of claims 1 to 12, wherein the clustering rules include rules for powering on and powering off the clusters.

14. The system according to any one of claims 1 to 13, wherein the service template defines a cluster manager to be deployed by the controller to manage the cluster.

15. The system according to any one of claims 1 to 14, wherein the controller constitutes the cluster via an out-of-band management connection.

16. The system according to any one of claims 1 to 15, wherein the cluster is deployed on a plurality of computing resources connected to each other via network resources.

17. The system according to any one of claims 1 to 16, wherein the controller employs a pivot_root process based on the service template, and the pivot_root process enables the cluster to be booted from a storage area network (SAN) without BIOS dependency.

18. The system according to any one of claims 1 to 17, wherein the system further comprises a plurality of resources, and the controller is configured to perform a plurality of operations on which the clusterable service is automatically scaled up to a cluster of the service based on the service template, the operations comprising: (1) reading the clustering rules; (2) deploying the service instances to the plurality of resources in accordance with the read clustering rules; (3) connecting the service instances as the cluster based on the read clustering rules; and (4) configuring the scheduler for managing the cluster based on the read clustering rules, the management of the cluster by the scheduler comprising scheduling tasks for the service instances in the cluster.

19. The system according to claim 18, wherein the system further includes an API endpoint associated with the service template, and the clustering rule (1) identifies a replacement API endpoint, and (2) defines the conditions under which the replacement API endpoint is invoked.

20. A computer reads multiple clustering rules from a service template, The steps include deploying multiple applications as a cluster using the aforementioned computer, in accordance with the aforementioned clustering rules, The steps include deploying a scheduler for managing the application using the aforementioned computer, Methods that include...

21. A computer program comprising a plurality of processor-executable instructions residing on a non-transient computer-readable storage medium, wherein the instructions are configured for execution by a processor, and the computer program causes the processor to automatically cluster clusterable services using a plurality of clustering rules defined by a service template, deploying a plurality of service instances as a cluster of services corresponding to the clusterable services, and causing the processor to deploy a scheduler for managing the service instances.

Images