Managing applications in a cluster

Abstract

A method for managing applications in a cluster is described. In one example, a first agent can be executed on a first programmable network adapter card installed in a first computing node in the cluster. The first agent can isolate the application executed on the first computing node. Thereafter, the application can be managed by a second computing node.

Description

Technical Field

[0001] The present disclosure generally relates to managing applications in a cluster. Background Art

[0002] Clusters allow different computing devices, called nodes, to operate together to act as a unified computing resource. These types of architectures feature data centers with multiple computing nodes or servers for handling various workloads. Computing nodes are able to perform various computing functions, such as I / O packet processing or workload management, without loading the core processing unit of the computing node. In addition, computing nodes can also monitor and manage faults or errors to ensure the continued operation or availability of computing nodes over a longer period of time. Summary of the invention

[0003] According to one aspect of the present disclosure, a processing system is provided, comprising: at least one processor; and a non-transitory machine-readable storage medium, the machine-readable storage medium comprising instructions executable by the at least one processor to perform the following operations: sending control information from the system to a first programmable network adapter in a first computing node via a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receiving the information message from the first agent executed on the first programmable network adapter at the system in response to the control information, wherein the information message is related to a program executed on the first computing node; determining a second periodicity for the first agent to send the information message to the system via the network; comparing the determined second periodicity with the first periodicity specified in the control information; moving the program from the first computing node to a second computing node based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; and sending instructions to a second agent executed on a second programmable network adapter in the second computing node to manage the program on the second computing node.

[0004] According to another aspect of the present disclosure, a method for failover is provided, comprising: sending control information from a system to a first programmable network adapter in a first computing node via a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receiving the information message from the first agent executed on the first programmable network adapter at the system in response to the control information, wherein the information message is related to a program executed on the first computing node; determining, by the system, a second periodicity for the first agent to send the information message to the system via the network; comparing the determined second periodicity with the first periodicity specified in the control information; moving, by the system, the program from the first computing node to a second computing node as part of a failover process based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; and sending, by the system after the failover process, instructions to a second agent executed on a second programmable network adapter in the second computing node to manage the program on the second computing node.

[0005] According to another aspect of the present disclosure, a non-transitory computer-readable medium comprising instructions is provided, which when executed causes a system to: send control information from the system to a first programmable network adapter in a first computing node via a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receive the information message from the first agent executed on the first programmable network adapter at the system in response to the control information, wherein the information message is related to a program executed on the first computing node; determine a second periodicity for the first agent to send the information message to the system via the network; compare the determined second periodicity with the first periodicity specified in the control information; based on determining that the determined second periodicity is greater than the first periodicity specified in the control information, move the program from the first computing node to a second computing node as part of a failover process; and send instructions to a second agent executed on a second programmable network adapter in the second computing node to manage the program on the second computing node. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Systems and / or methods according to examples of the present subject matter are now described by way of example and with reference to the accompanying drawings, in which:

[0007] Figure 1 A system for managing applications in a cluster according to an example is illustrated;

[0008] Figure 2 illustrates a computing environment for implementing a cluster according to an example;

[0009] Figure 3 An evaluation system for managing applications in a cluster according to an example is illustrated;

[0010] Figure 4 A method for managing applications in a cluster according to an example is illustrated;

[0011] Figure 5-6 An example method for managing applications in a cluster is illustrated; and

[0012] Figure 7 A system environment implementing a non-transitory computer-readable medium for managing applications in a cluster according to an example is illustrated. DETAILED DESCRIPTION

[0013] In a cluster-based architecture system, a group of computing nodes can be logically grouped together to form a computing cluster. The computing workload can be distributed among the computing nodes that can communicate with each other. The computing nodes can be physical servers or can be systems that implement a single server, one or more applications, multiple virtual machines, or different combinations thereof in turn.

[0014] The computing nodes within the cluster can be configured to process requests that they may receive from another computing device and provide services to the requesting computing device. The services provided by the computing nodes may include providing access to data, implementing various operations, or a combination thereof. For example, a computing node may receive a request from a client device, process the request, and provide a response to the client device accordingly. In order to process the request, the computing node may access a storage system, and retrieve and process data to provide a response. The computing node may also include different applications that can access the storage system (either implemented on the computing node itself or implemented in a virtual environment running on the computing node).

[0015] In addition to servicing requests received from other computing devices, compute nodes may also implement functions related to managing the lifecycle of applications hosted therein. These functions include functions such as launching applications and monitoring their operation to assess whether the application (or compute node) is running in a high-performance manner. Applications may be monitored to ensure that continuous operation or uptime states, as opposed to downtime states for applications or compute nodes, continue for extended periods of time. This state of the cluster is referred to as high availability.

[0016] Enabling high availability involves monitoring, evaluating, and minimizing instances of downtime to ensure continuous availability. If a compute node is no longer available, e.g., no longer able to service requests, a failover operation can be performed during which applications executing on the failed node can be moved to other compute nodes. In these instances where errors occur, ensuring high availability involves finding another compute node (e.g., within the cluster where the failed compute node resides) to which servicing of subsequent requests can be transferred.

[0017] Various functions that can be used to ensure high availability can be implemented by the operating system of a given computing node. In some cases, the mechanism for ensuring high availability may not be available at the hardware level of different computing nodes. In addition, computing nodes and storage systems can be used in different configurations to address different business considerations. The mechanism for achieving high availability, if implemented at the operating system level, may not take into account the applications that can be implemented within a given computing node. For example, an application can be implemented to run on the operating system of a computing node, or can be implemented in a virtualized environment. When using a high availability mechanism, this change in the way an application can be implemented may pose a challenge. As a result, a solution for providing high availability that can be implemented at the platform level (i.e., at a computing node) is not available.

[0018] Regardless of the manner in which this approach is used, the approach used to achieve high availability may load the computing resources of the computing nodes, which may affect the computing nodes' service of requests. Recent developments have led to the installation of programmable network adapter cards (also known as intelligent network interface cards) on computing nodes. Such programmable network adapter cards not only provide interfaces to enable communication between computing nodes within a cluster-based computing environment, but also provide processing power for functions such as storage management and security functions. It is worth noting that such functions can be implemented at the computing node level independent of the operating system.

[0019] Methods for managing applications in a cluster-based computing environment are described. In one example, a cluster can be implemented in a data center including multiple computing nodes. In one example, a computing node can be a server that implements multiple applications. In another example, a computing node can be a server that implements multiple virtual machine instances, which further implement respective applications. A computing node can be one of the nodes in a cluster. A cluster can be considered as a group of computing nodes, each of which can operate and run independently of each other. The resulting cluster can be considered as a unified computing resource for performing computing operations. Each computing node in the cluster can be coupled to a storage system to access and perform read and write operations on data stored in the storage system. The operation of the cluster can be monitored to assess the occurrence of failures and take remedial measures to ensure high availability.

[0020] A manner of assessing the occurrence of a fault is described in the example context of a first computing node and a second computing node within a given cluster. In one example, the first computing node may include a first programmable network adapter card (referred to as a first network card). In a similar manner, the second computing node includes a second network card. Examples of such programmable network adapter cards include intelligent network interface cards (or intelligent NICs) that can be installed within a computing node of a cluster. As will be discussed, the use of a first network card and a second network card to assess the occurrence of a fault (which in turn can provide high availability) achieves many technical advantages. For example, computing resources, such as a processor, of a computing node can be offloaded using the first network card and the second network card. In addition, the assessment of the occurrence of a fault by the first network card and the second network card at the computing node level is independent of the operating system. This approach is also not affected by the manner in which applications are deployed on computing nodes.

[0021] Both the first network card and the second network card may include processing resources that can be used to offload certain processing functions from the local processors of the corresponding first computing node and the second computing node. In an example, the first network card and the second network card may include a first agent and a second agent, respectively. As will be further described, the agent with the network card will implement the application in the management cluster and thereby ensure high availability.

[0022] In one example, the cluster may include a central evaluation system that may communicate with the first computing node and the second computing node. In one example, the central evaluation system may receive information messages from the first agent and the second agent related to the operation of the first computing node and the second computing node. The information messages may be related to certain operational or functional aspects of the first computing node and the second computing node. For example, the information may be related to health information of the first computing node and the second computing node, or may include information related to applications that may be executed on the first computing node and the second computing node.

[0023] The information messages received from the respective agents of the first network card and the second network card can be processed to determine whether a failure has occurred in either of the first computing node and the second computing node. For example, based on the processing of the data within the information message, the central evaluation system can ascertain that the first computing node (or one of the applications on the first computing node) is experiencing a failure, while the second computing node is operating in a high-performance manner. The determination can be based on a plurality of conditions that can be predefined. For example, the determination can be based on a change in the frequency of messages, certain information included in the messages, or the absence of any messages.

[0024] Once it is determined that the first computing node is experiencing a failure, a second agent within a second network card of the second computing node can be notified. Thereafter, the second agent within the second network card can cause the application to be moved to the second computing node (e.g., perform a failover). In one example, the first agent can initially isolate the application within the first computing node from the storage system, wherein once isolated, the application can be initialized to execute on the second computing node. It can be noted that by using resources associated with the first network card to monitor the applications and operations of each computing node, high availability is provided without burdening the operating systems of the computing nodes within the cluster. In addition, a central evaluation system within the cluster handles failovers locally, making the failover process more efficient.

[0025] The above examples are further described in conjunction with the accompanying drawings. It should be noted that the description and drawings illustrate only the principles of the present subject matter. It will therefore be understood that various arrangements embodying the principles of the present subject matter, although not explicitly described or shown herein, may be designed from the specification and are included within the scope of the specification. In addition, all statements herein that reference the principles, aspects, and examples of the present subject matter and specific examples thereof are intended to encompass their equivalents. In the accompanying drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The same numbers are used throughout the drawings to represent the same features and components.

[0026] Figure 1 The diagram shows the example for evaluating the cluster ( Figure 1 1. The system 102 includes a processor (or processors) 104 and a machine-readable storage medium 106 coupled to and accessible by the processor (or processors) 104. The machine-readable storage medium 106 stores instructions (or instructions) 108 accessible by the processor (or processors) 104. In one example, the system 102 may be implemented as a standalone computing device that may be used in conjunction with a cluster (or clusters). Figure 1 1 and 2. The processors 104 may communicate with computing nodes (such as a first computing node and a second computing node) within the processor 104 (not shown). The processors 104 may be implemented as dedicated processors, shared processors, or multiple individual processors, some of which may be shared.

[0027] The machine-readable storage medium 106 can be communicatively connected to the (multiple) processors 104. Among other capabilities, the (multiple) processors 104 can obtain and execute (multiple) computer-readable instructions 108 stored in the machine-readable storage medium 106. In operation, the (multiple) processors 104 can execute instructions 110 to obtain information from a first agent executed in a first network card. As used herein, the first agent (or other agents) can be implemented using programmable instructions. For example, such programmable instructions, when executed, may generate processes that can be used to perform various functions, such as networking, storage, and security functions. For example, an agent in a network card can be used to monitor operating parameters of a computing node in a cluster computing environment. In one example, the agent can be executed and controlled by a processing element of each network card (e.g., a first network card).

[0028] The first network card is then installed in a first computing node within the cluster. The information obtained as a result of executing instruction 110 can be related to an application that can be executed in the first computing node and can be communicatively coupled to the storage system. In one example, the information can be related to the operation and health of the application executed in the first computing node.

[0029] Once the system 102 receives the information, the processor(s) 104 may execute instructions 112 to cause the first agent of the first network card to isolate the application from the storage system to which it is coupled. As used herein, "isolation" refers to preventing the application from accessing the storage system (e.g., a LUN). The isolation of the application from the storage system may be in response to processing the information about the application received from the first agent. In one example, the isolation may restrict the application that may be experiencing an adverse operating condition from accessing any storage system that the first computing node may communicate with. In other words, once the application is isolated, the isolated application may not be able to perform any read or write operations on the storage system.

[0030] In one example, system 102 may determine that the first computing node is experiencing a failure based on processing the information. Based on determining this, instructions 112 may cause the first agent to isolate the application from the storage system. When isolated, the application or the first computing node may be constrained to prevent it from accessing the storage system. In one example, the application or the first computing node may be constrained to prevent it from accessing a network or any other service that was otherwise available to the first computing node before isolation.

[0031] In the case where the application is isolated from the storage system, the processor(s) 104 may further execute instructions 114 to transmit the instructions to the second agent. In one example, the second agent may be executed on a second network card installed in a second computing node in a cluster, and the first computing node may exist in the cluster. In one example, the transmitted instructions may enable the second agent to manage the application when executed. In one example, the second agent may start the application, and the application may then begin to run and operate on the second computing node.

[0032] Figure 2 The diagram illustrates a computing environment for implementing cluster 200. In one example, cluster 200 may be implemented in a data center. As will be described, various approaches implemented in cluster 200 will achieve high availability of computing nodes in cluster 200. In one example, cluster 200 may include a first computing node 202 and a second computing node 204. First computing node 202 may be implemented as a server device or an independent computing system. First computing node 202 may also include a processor 206 and an interface 208. In one example, processor 206 is similar to processor(s) 104 (e.g., Figure 1 200 ). Interface 208 may include software-implemented interfaces and hardware-implemented interfaces. Interface 208 enables first computing node 202 to be communicatively coupled to other computing nodes within any given cluster. Interface 208 may also enable components of first computing node 202 to communicate with each other. First computing node 202 may also include applications 210-1, 2, ..., N. Applications 210-1, 2, ..., N (collectively referred to as applications 210) may be executed on first computing node 202. The execution of any one application 210 or a combination of multiple applications 210 may implement various services within cluster 200. In one example, application 210 may be executed on operating system 212.

[0033] It can be noted that the application 210 can be run in the first computing node 202 in different combinations. For example, the application 210 can be implemented and executed within a virtual environment (e.g., within a virtual machine). In this case, the virtual machine may be running a single or multiple applications, or in other cases, multiple virtual machines may be running a single or multiple applications. In other examples, the application 210 can be deployed in a container or can be installed on an operating system (such as operating system 212) of the first computing node 202.

[0034] Within the first computing node 202, a programmable network adapter card 214 is installed. Examples of programmable network adapter cards 214 include intelligent network interface cards (NICs). Network cards such as programmable network adapter cards 214 enable computing devices to communicate and couple with other computing devices or network devices within a network. In some instances, a network card may also enable a computing device to be communicatively coupled to another network. Programmable network adapter cards 214 differ from network cards in that additional computing, networking, storage, and security functions are implemented through processing elements provided therein rather than the main processor of the computing device. For example, a programmable network adapter card 214 (referred to as a first network card 214) may include a processing element that can implement certain functions instead of a processor 206. These functions may be implemented by circuits or by software-defined logic.

[0035] In one example, the first network card 214 may include a controller (for enabling networking functions), a processor unit, and interfaces that may communicatively couple the first computing node 202 to other computing nodes or other types of devices (such as storage systems). The first network card 214 may also include additional components that may enhance certain computing functions of the first network card 214 (such as processing power or storage).

[0036] The first network card 214 may also include a first agent 216 and other engines 218. As described later, the first agent 216 will monitor the health of applications that can be executed on the first computing node 202. Other engines 218 can be used to perform other functions by the first network card 214. Examples of such functions may include functions of physical devices or may include functions related to other functional features of the first computing node 202. These features may include functions related to networking, security, and availability. To this end, other engines 218 can implement a set of physical functions (physical function, PF) or virtual functions (virtual function, VF). A physical function (PF) refers to a physical device (such as a first network card or a second network card), while a virtual function (VF) refers to a virtual instance of a PF or a device. In this way, a physical device can appear as multiple virtual instances or virtual functions and can be used as the multiple virtual instances or virtual functions. In an example, a PF can be considered to provide access and control to certain core functions (such as the first network card 214 and the second network card 232). A PF can be implemented by a driver, which in turn may need to be docked with a network card. On the other hand, VF can also realize the transmission of data or information between the network card and the application that can be executed on any given computing node. For example, VF can be associated with one of the applications 210 and can manage the application and the storage system 222 to communicate data.

[0037] The first agent 216 and other engines 218 (collectively referred to as engines) can be implemented as a combination of hardware and programming (e.g., programmable instructions for implementing various functions). In the examples described herein, the combination of hardware and programming can be implemented in several different ways. For example, the programming of the engine can be executable instructions. These instructions can be stored on a non-transient machine-readable storage medium, which can be directly coupled or indirectly coupled (e.g., via a networked manner) to the first computing node 202. In an example, the engine can include processing resources (e.g., a single processor or a combination of multiple processors) to execute instructions. In this example, the non-transient machine-readable storage medium can store instructions (such as instructions 208) that implement the engine when executed by the processing resources. In other examples, the engine can be implemented as an electronic circuit. The first computing node 202 can also include a port 220, through which the first computing node 202 can be coupled to a storage system 222. Examples of storage system 222 can include, but are not limited to, physical storage (such as a hard disk, etc.), virtual storage (such as a distributed file system, etc.), and a logical unit number (logical unit number, LUN).

[0038] The second computing node 204 may include components similar to those of the first computing node 202. For example, the second computing node 204 may include a processor 224 and an interface 226. The second computing node 204 may also include an application 228 (which may be similar to the application 210), wherein the application 228 runs on an operating system 230. Further, the second computing node 204 may include a second network card 232. The second network card 232 (similar to the first network card 214) may be an intelligent new interface card. The second network card 232 also includes a second agent 234 and other engines 236. The second agent 234 and other engines may be similar to engines (e.g., the first network card 214 and other engines 218 in the first computing node 202). In an example, the second computing node 204 may include a port 238 that connects the second computing node 204 to a storage system 240.

[0039] As previously described, the first computing node 202 and the second computing node 204 are part of the cluster 200. The cluster 200 thus formed can operate as a unified computing resource to implement a service through the combined operation of the first computing node 202 and the second computing node 204. Although not depicted, the cluster 200 may include other computing nodes ( Figure 2 Even in these examples, these approaches may be implemented for such other computing nodes without departing from the scope of the present subject matter.

[0040] Further, both the first computing node 202 and the second computing node 204 can be communicatively coupled to a central evaluation system 242. In one example, the central evaluation system 242 (referred to as evaluation system 242) can be similar to the system 102 (eg, Figure 1 ). The evaluation system 242 may further include an availability agent 244. As will be described, the availability agent 244 may further assist in providing services with high availability through computing nodes (such as the first computing node 202 and the second computing node 204). In one example, the evaluation system 242 may be coupled to the first computing node 202 and the second computing node 204 through a network. The network may be a private network or a public network, and may be implemented as a wired network, a wireless network, or a combination of a wired and wireless network. The network may also include a set of separate networks that are interconnected and used as a single large network, such as the Internet. Examples of such separate networks include, but are not limited to, the Global System for Mobile Communication (GSM) network, the Universal Mobile Telecommunications System (UMTS) network, the Personal Communications Service (PCS) network, the Time Division Multiple Access (TDMA) network, the Code Division Multiple Access (CDMA) network, the Next Generation Network (NGN), the Public Switched Telephone Network (PSTN), the Long Term Evolution (LTE), and the Integrated Services Digital Network (ISDN).

[0041] The operation of the evaluation system 242 will be combined with Figure 3 Further description. Figure 3 Various components of an example evaluation system 242 are depicted. Figure 3 , the assessment system 242 includes a central availability agent 244. In addition, the assessment system 242 may include (multiple) instructions 302. In an example, the (multiple) instructions 302 may be retrieved from the memory of the assessment system 242 and executed by its processor to call the availability agent 244. The assessment system 242 may also include a control strategy 304, node information 306, and application information 308.

[0042] Return to Figure 2 In the example cluster 200 depicted in FIG. 2 , applications 210 may be executed and run on a first computing node 202 to provide services. When they are running, they may be accessing a storage system 222 to perform read and write operations on data stored in the storage system 222. In one example, a first network card 214 may initialize a first agent 216. The initialization of the first agent 216 may be based on a control policy 304. For example, the control policy 304 may specify which monitoring functions the first agent 216 is to implement. In addition, the control policy 304 may also specify which applications 210 are to be monitored and which parameters of the application 210 are to be monitored. In one example, the control policy 304 may be specified by an evaluation system 242. In a similar manner, the evaluation system 242 may also specify other control policies 304 for a second agent, such as an agent 234 within a second computing node 204. The control policy 304 may be specified for an application 210 or a virtual instance within which the application 210 may run. In another example, the control policy 304 may also be specified for a container environment within which the application 210 may be implemented.

[0043] Once the first agent 216 and the second agent 234 are initialized, they may transmit their identification information to the assessment system 242. The identification information may be used by the assessment system 242 to discover and identify different computing nodes that may be present within a cluster (e.g., cluster 200). In this example, based on the identification information received from the first agent 216 and the second agent 234, the assessment system 242 may determine that the first computing node 202 and the second computing node 204 are present within the cluster 200. In an example, the information identifying the different computing nodes (i.e., the first computing node 202 and the second computing node 204) may be stored in the node information 306. It may be noted that although this example is described with respect to the first computing node 202 and the second computing node 204, the discovery and identification process described may be applied to any number of computing nodes that may be present within the cluster 200. In an example, the instruction(s) 302 may be executed to initialize the first agent 216 and the second agent 234.

[0044] With the first agent 216 now initialized, it can now collect information related to the application 210. In one example, the first agent 216 can collect information by interfacing with the application layer through the other engine 218. Information from the application 210 can be collected at predefined intervals. The intervals can in turn be dictated by the control policy 304.

[0045] Once the information is collected from the application 210, the first agent 216 can transmit the information as a message to the evaluation system 242. The information message received by the evaluation system 242 can then be stored as application information 308. In an example, the information message can provide various information related to the first computing node 202 (or any other computing node within the cluster 200). Examples of such information include, but are not limited to, the identity of the first computing node 202, information related to the application 210, information about the computing resources of the first computing node 202, and the health and operating parameters of the first computing node 202. In another example, the information message can be a heartbeat message generated and transmitted by the first computing node 202. In a similar manner, the second computing node 204 can also generate such a heartbeat message to send to other computing nodes within the cluster 200. The heartbeat message can be considered as a message that can be shared periodically between the various computing nodes in the cluster 200.

[0046] The availability agent 244 within the assessment system 242 can then determine whether the computing node under consideration (e.g., the first computing node 202) is experiencing any problems. To this end, the availability agent 244 can monitor the first computing node 202 and its application 210 in two stages. In the first stage, the availability agent 244 can determine the periodicity or predefined time period at which the first agent 216 transmits information messages. The periodicity can be considered as a regular time interval at which information messages can be transmitted. For example, if the periodicity is x seconds, the first agent 216 can transmit information messages every x seconds.

[0047] The periodicity followed by the first agent 216 for transmitting information messages can in turn be specified by the control policy 304. It can be noted that any additional or unplanned load or problem may cause congestion at the first computing node 202. If these instances of congestion occur, it is possible that the periodicity of incoming information messages may increase. On the other hand, if the first computing node 202 is operating in a high-performance mode, the periodicity of the incoming messages can remain consistent with the periodicity that may have been specified by the control policy 304.

[0048] The availability agent 244 can continue to recursively monitor the periodicity of the incoming messages. If the periodicity of the message information increases to exceed a certain threshold limit, the availability agent 244 can ascertain that the first computing node 202 may be experiencing a condition that may correspond to the onset of an increased load or the occurrence of an error. For example, an increase in the time interval between each consecutive information message can indicate a variety of conditions, namely, an application crash, an agent crash, or, in an instance involving a virtual machine, a crash of the virtual machine itself. In any case, any delay in receiving an information message or a failure in receiving an information message may be attributed to any one or more applications 210 being rendered inactive.

[0049] To this end, the availability agent 244 may compare the periodicity of the incoming message information and compare it to the periodicity specified under the control policy 304. In the event that the periodicity is determined to have increased to a value greater than a predefined threshold, the instruction(s) 302 may be executed to further cause the availability agent 244 to process the application information 308 (which in turn is derived based on the incoming information message). In one example, the availability agent 244 may process the application information 308 to determine various operating parameters related to the application 210 and the first computing node 202. The operating parameters determined from the application information 308 may then be compared to predefined values ​​or thresholds to determine whether a given node (i.e., the first computing node 202) is unavailable. Accordingly, the availability agent 244 may determine whether to implement remediation procedures, which include failover to ensure high availability.

[0050] It may be noted that possible candidate computing nodes on which a failover may be performed may be determined based on information messages shared by the various computing nodes. In the context of this example, a second agent 234 within the second computing node 204 may periodically transmit information messages to an availability agent 244 within the evaluation system 242. The availability agent 244 may then determine the operational health of the second computing node 204. If found to be acceptable, the orchestration engine 246 may initiate a failover process and move the application 210 to the second computing node 204. In an example, the availability agent 244 may communicate with the orchestration engine 246 to determine which of the first computing node 202 and the second computing node 204 may be suitable for failover of the application.

[0051] As described above, the second computing nodes 204 may transmit their information messages to the evaluation system 242. In a similar manner, the other computing nodes ( Figure 2 204) may also transmit their respective information messages to the availability agent 244 of the evaluation system 242. The availability agent 244 may evaluate the operational status of other nodes (e.g., the second computing node 204) to determine one of these computing nodes as a candidate computing node to which the application 210 may be moved.

[0052] In one example, one of the agents executing on each programmable network adapter card may be selected as the coordination agent. The selection of a given computing node as the coordination agent may be based on multiple conditions. For example, it may depend on the computing properties of the programmable network adapter card on which the given agent may be executing. In this case, different agents may initially share information messages between them. In one example, the information message may include an identification message. The identification message may additionally provide an identifier corresponding to the first network card 214. In another example, the identification message may also indicate information related to the computing resources of the first network card 214.

[0053] Once information is exchanged, it can be processed to identify the agent executed on the programmable network adapter card with the maximum computing resources. The identified agent can then be selected or specified as a coordination agent. Once appointed, the selected coordination agent can collect information messages from other agents in other computing nodes and transmit them to an availability agent, such as an availability agent 244 running on the evaluation system 242. It can be understood that the evaluation system 242 can therefore receive information messages from other agents, or can receive information messages of other computing nodes by the coordination agent. The coordination agent can be merged with other agents to form a sub-cluster. In another example, a plurality of other agents in other computing nodes can be designated as coordination agents. Other coordination agents can be merged with a group of other agents to form other sub-clusters. In these examples, different agents can be organized into multiple groups, each group has a respective selected coordination agent, and the selected coordination agent then transmits information messages to a central evaluation system, such as the evaluation system 242. These and other examples still fall within the scope of this theme.

[0054] As previously described, orchestration engine 246 can enable failover to a different node (referred to as a target node) upon determining that a given computing node is experiencing a failure or error. In the example described above, first computing node 202 is described as experiencing a failure based on application information 308 processed by availability agent 244. For purposes of illustration, this example is described assuming that second computing node 204 is determined to be a suitable candidate for failover in the event that first computing node 202 is no longer available.

[0055] Once the second computing node 204 is identified (i.e., the target node 204), the orchestration engine 246 can initialize the isolation of the first computing node 202 and the corresponding storage system 222. To this end, the orchestration engine 246 can transmit instructions to the first agent 216 to isolate any one application 210 (e.g., the application 210-1 that is determined to be operating incorrectly) from accessing the storage system 222. In an example, the first agent 216 running on the first network card 214 can isolate the application 210-1 from accessing any other service or device that may communicate with the first computing node 202. In this case, other engines 218 that implement multiple virtual functions (VFs) can be controlled and managed by the first agent 216. The first agent 216 can terminate the VF that manages access to the storage system 222. Once the VF is terminated, access to the storage system 222 is lost, resulting in the isolation of the application 210-1 from the storage system 222. In the case where the application 210 is implemented in a virtual machine or container, the virtual machine or container can stop accessing a network to which the first computing node 202 can be connected. In one example, the virtual machine or container (as the case may be) can stop communicating with the storage system 222. The storage system 222 can manage the storage LUNs implemented on the first computing node 202. It can be noted that other approaches for isolating the application 210-1 or the first computing node 202 can be employed without departing from the scope of the present subject matter. In one example, the execution of the application 210-1 can be terminated.

[0056] Once the application 210-1 is isolated, configuration information related to the isolated application 210-1 may be retrieved. In one example, the configuration information may be obtained from the application information 308. The configuration information of the application 210-1 may define various parameters based on which the application 210-1 initially accesses any of the storage systems 222, services, or other devices within the cluster 200. Examples of such configuration information include, but are not limited to, network configurations, port numbers, and the like.

[0057] The orchestration engine 246 can then use the configuration information thus collected to prepare network connections to other storage systems, services, or other devices. Once the network connections and other configurations are defined, the orchestration engine 246 can begin the startup of the application 210-1, but on the target computing node 204. In one example, the orchestration engine 246 can send a startup instruction to the second agent 234 (now referred to as the target agent 234). The target agent 234 can initialize or start the application 210-1 (as the application 228-1) when receiving the instruction from the orchestration engine 246 of the evaluation system 242. The application 228-1 can then begin accessing the storage system, service, or other device based on the configuration information. For example, when the application 228-1 is executed, it can now access the storage system 240 through the port 238.

[0058] The first agent 216 can take proactive measures to ensure the availability of the first computing node 202. As previously described, the first agent 216 can periodically send information messages to the availability agent 244 of the evaluation system 242. The availability agent 244 can generate a confirmation message and send the confirmation message in response to the information message received from the first agent 216. Since the confirmation message is generated in response to the information message, the periodicity of the confirmation message will correspond to the periodicity of the information message. In an example, the first agent 216 can monitor any changes in the periodicity of the confirmation messages received from the availability agent 244. If the periodicity (i.e., the time interval between consecutive incoming confirmation messages) increases, the first agent 216 can determine that the first computing node 202 may be experiencing congestion, and therefore a delay between consecutive confirmation messages may have occurred. Therefore, the absence of any confirmation message may indicate that the first computing node 202 may have become inoperable and no longer communicate with the evaluation system 242.

[0059] Upon detecting the above situation, the first agent 216 can initiate the isolation of the first computing node 202. To this end, the first agent 216 can initiate the termination of one of the applications (e.g., application 210-1), which cannot run normally or has experienced a failure. Once execution is terminated, the application 210-1 also cuts off any communication with the first network card 214 within the first computing node 202. Once the communication with the first network card 214 has ended, the application 210-1 may no longer be able to access any network, storage system (e.g., storage system 222) or any other device. In an example, the availability agent 244 can also take additional steps to implement a failover to an appropriate computing node (such as the second computing node 204). It can be noted that the isolation initiated by the first agent 216 can prevent any access attempts of the application 210-1, and the availability agent 244 can implement a failover action to move the application 210-1 to the second computing node 204. In an example, the application 210-1 can be initiated or started on the second computing node 204 as the application 228-1. Within the second computing node 204, the second agent 234 can transmit the application 228-1 when it is detected to the evaluation system 242. In one example, upon receiving notification that the application 210-1 (now the application 228-1) has moved to the second computing node 204, the availability agent 244 can clear the records and configuration information related to the application 210-1 when the application 210-1 is running on the first computing node 202.

[0060] In one example, a fault can be found in an instance where there is a communication interruption between various agents and the coordination agent. To illustrate this example, it is assumed that the second agent 234 is a coordination agent. During the high-performance operation of the first computing node 202 and the second computing node 204, the second agent 234 (acting as a coordination agent) can collect information messages from the first agent 216 and transmit them to the evaluation system 242. If the first computing node 202 encounters an error, it may cause the first agent 216 to be unable to share the information messages of the first computing node 202 with the second agent 234. The second agent 234 can then transmit the situation of missing any incoming information messages to the availability agent 244 of the evaluation system 242. The availability agent 244 can then send instructions to the first agent 216 to isolate the application or the first computing node 202. In addition, the availability agent 244 can also send instructions to the second agent 234 for starting the failover of the application 210 on the first computing node 202 to the second computing node 204.

[0061] As described above, the first network card 214 and the second network card 232 can be used to achieve high availability of various applications that can be run thereon. Also as described above, the first network card 214 and the second network card 232 can communicate with the application through a driver that can be installed on the operating system 212, 230. In the case of a virtual machine or container, the first network card 214 and the second network card 232 can communicate with the application 210 through a protocol based on the single root input / output virtualization (SR-IOV) specification. It can be noted that the SR-IOV specification is just an example - other examples are possible without departing from the scope of the present subject matter.

[0062] Figure 4 A method 400 for handling a failover in a distributed computing system according to another example is illustrated. The order in which the above methods are described is not intended to be interpreted as a limitation, and some of the method boxes described may be combined in different orders to implement these methods or alternative methods. In addition, the above methods may be implemented in any suitable hardware, computer-readable instructions, or a combination thereof. The steps of these methods may be performed by the system under the instructions of machine-executable instructions stored on a non-transient computer-readable medium, or by a dedicated hardware circuit, a microcontroller, or a logic circuit. For example, method 400 may be performed by a first agent 216 executed on a first network card 214 installed in a first computing node 202. Here, some examples are also intended to cover non-transient computer-readable media, such as digital data storage media, which are computer-readable and encode computer-executable instructions, wherein the instructions perform some or all of the steps of the above methods.

[0063] At block 402, a first identification message indicating an identity and attributes of a first programmable network adapter card may be transmitted by a first agent. The first agent may be executed on a first programmable network adapter card, wherein the network card is installed on a first computing node. For example, the first agent 216 may transmit the first identification message to a second agent 234 executing in a second network card 232 of a second computing node 204. The first identification message may provide an identifier of the first network card 214 and computing attributes. The identification message is shared with the second agent 234 executing in the second network card 232.

[0064] At block 404, similar to the first identification, a second identification can be received from the second agent. For example, the first agent 216 receives a second identification message transmitted from the second agent 234 of the second computing node 204. Similar to the first identification message, the second identification message can provide an identifier of the second network card 232 and computing attributes.

[0065] At block 406, one of the first agent and the second agent may be designated as a coordination agent based on the first identification message and the second identification message. Once one of the computing nodes is designated as a coordination agent, the coordination node may then obtain information messages from other agents executing on other network cards within the cluster. For example, based on a comparison of the computing resources of the first network card 214 and the second network card 232, the first agent 216 may be designated as a coordination agent. Once designated as a coordination agent, the first agent 216 may collect information messages from other agents (such as the second agent 234) and transmit the collected information messages to the availability agent 244 of the evaluation system 242.

[0066] Figure 5 A method 500 for handling failover in a distributed computing system according to another example is illustrated. Similar to the method 400, the method 500 may also be implemented by one of the first computing node 202, the second computing node 204, and the evaluation system 242. As described above, the first computing node 202 includes a first network card 214 running a first agent 216. In a similar manner, the second computing node 204 may include a second network card 232 running a second agent 234. Although depicted as including only two computing nodes, the cluster 200 may include additional computing nodes without departing from the scope of the present subject matter.

[0067] At box 502, a set of control policies can be obtained. For example, control policies 304 can be obtained from the evaluation system 242. Control policies 304 can specify what monitoring functions are to be implemented by agents (such as the first agent 216) within various programmable network adapter cards. In addition, control policies 304 can also specify which of the applications 210 (running on the first computing node 202) are to be monitored and which parameters of the applications 210 are to be monitored. In a similar manner, the evaluation system 242 can also specify other control policies 304 for the second agent (such as the agent 234 within the second computing node 204). Control policies 304 can be specified for the application 210 or for the virtual instance in which the application 210 may be running. In another example, control policies 304 can also be specified for the container environment in which the application 210 can be implemented.

[0068] At block 504, the first agent and the second agent may be initialized. For example, the first agent 216 and the second agent 234 may be initialized by the first network card 214 and the second network card 232, respectively. In the context of this example, the first agent 216 may be implemented within the first network card 214, wherein the first network card 214 is implemented within the first computing node 202. In a similar manner, the second agent 234 may also be initialized by the second agent 234 implemented on the second network card 232 installed within the second computing node 204. The first computing node 202 and the second computing node 204 may be computing nodes within the cluster 200. The first computing node 202 may provide services using the application 210. When they are running, they may be accessing the storage system 222 to perform read and write operations on data stored in the storage system 222. In an example, once the first agent 216 and the second agent 234 are initialized, they may begin to monitor the operations of the first computing node 202 and the second computing node 204 according to the control policy 304.

[0069] At block 506, identification information of the computing nodes may be transmitted to a central evaluation system. The central evaluation system may communicate with different computing nodes within the cluster. For example, the first agent 216 and the second agent 234 may send identification information of the first computing node 202 and the second computing node 204 to the evaluation system 242. Based on the identification information, the evaluation system 242 may determine that the first computing node 202 and the second computing node 204 exist within the cluster 200. In an example, the information identifying the first computing node 202 and the second computing node 204 may be stored in the node information 306.

[0070] At block 508, the agent may monitor and collect information related to the application running on the computing node. For example, the first agent 216 collects information about the application 210. In one example, the first agent 216 may collect information by interfacing with the application layer through the other engine 218. Information from the application 210 may be collected at predefined intervals. The intervals may in turn be specified by the control policy 304.

[0071] At block 510, information related to different applications is transmitted to the evaluation system as periodic messages. For example, the first agent 216 may transmit the information as messages to the evaluation system 242. The information messages may be transmitted at a predefined periodicity, i.e., each successive information message is transmitted after a fixed time interval. The information messages received by the evaluation system 242 may then be stored as application information 308. In one example, the information messages may be shared by the first agent 216 and the second agent 234, wherein the first agent 216 and the second agent 234 share information about their respective computing nodes (such as the first computing node 202 and the second computing node 204). In one example, one of the agents (e.g., the first agent 216) may be selected as a coordination node. Thereafter, the first agent 216 may then collect the information messages from the second agent 234 and transmit them to the evaluation system 242. In an example, availability agent 244 of assessment system 242 may communicate with orchestration engine 246 to determine which of first computing node 202 and second computing node 204 is suitable for failover of application 210 - 1 (or any other application 210 that may be experiencing a failure).

[0072] In box 512, the information message received from the agent can be processed to determine whether the computing node is experiencing any problems. For example, the availability agent 244 can process the information message to determine whether, for example, the first computing node 202 is experiencing any problems. The availability agent 244 can initially determine whether the periodicity of the incoming information message has changed. For example, any additional or unplanned load or problem may cause congestion at the first computing node 202, so the first computing node 202 may eventually transmit information messages less frequently. The availability agent 244 can continue to monitor the periodicity of the incoming messages. If the periodicity of the message information increases to exceed a certain threshold limit, the availability agent 244 can find out that the first computing node 202 may be experiencing a condition that may correspond to the beginning of an increased load or the occurrence of an error.

[0073] At block 514, application information related to an application running on a given computing node may be processed. For example, availability agent 244 may process application information 308 (which in turn is derived based on the incoming information message) to determine operating parameters related to application 210 and first computing node 202. The operating parameters determined from application information 308 may then be compared to predefined values ​​or thresholds to determine whether a given node (i.e., first computing node 202) is unavailable.

[0074] At block 516, candidate nodes for moving the application may be determined. For example, availability agent 244 may determine the operational health of second computing node 204 (or any other computing node within cluster 200) based on the information message received from second agent 234. To this end, availability agent 244 may process node information 306 and application information 308 to determine the computing resources of second computing node 204 and evaluate whether the computing resources are able to support application 210. If found to be acceptable, orchestration engine 246 may initiate a failover process and move application 210 to second computing node 204.

[0075] At block 518, the application experiencing the problem may be isolated. For example, orchestration engine 246 may initialize isolation of first computing node 202 from corresponding storage system 222. To this end, orchestration engine 246 may isolate application 210-1 to prevent it from accessing storage system 222. In one example, application 210-1 may also be isolated to prevent it from accessing any other services or devices that may be in communication with first computing node 202. In one example, execution of application 210-1 may be terminated.

[0076] At block 520, configuration information related to the isolated application may be obtained. For example, once application 210-1 is isolated, availability agent 244 may retrieve configuration information related to isolated application 210-1 from application information 308. The configuration information may describe various parameters based on which application 210-1 initially accesses any of storage systems 222, services, or other devices within cluster 200. Examples of such configuration information may include, but are not limited to, network configuration, port numbers, and the like. Orchestration engine 246 may then use the configuration information so collected to prepare network connections to other storage systems, services, or other devices.

[0077] At block 522, the isolated application may be moved to an operational computing node. For example, once network connections and other configurations are defined for application 210, orchestration engine 246 may begin the startup of application 210-1, but on target computing node 204. In one example, orchestration engine 246 may send a startup instruction to second agent 234 (now referred to as target agent 234). Target agent 234, upon receiving instructions from orchestration engine 246 of assessment system 242, may initialize or start application 210-1 (as application 228-1). Application 228-1 may then begin accessing storage systems, services, or other devices based on the configuration information. For example, application 228-1, when executed, may now access storage system 240 through port 238.

[0078] Figure 6 A method 600 is illustrated, which can be implemented by an agent upon detecting a problem with a computing node on which it may be operating. The method 600 can also be implemented by one of the agents, such as the first agent 216 operating on the first network card 214 of the first computing node 202. To this end, the first agent 216 can take proactive measures to ensure the availability of the first computing node 202.

[0079] At block 602, the agent may periodically transmit information messages to the central assessment system. For example, the first agent 216 may periodically send information messages to the availability agent 244 of the assessment system 242. The information messages may provide information about the applications 210 that may be executed on the first computing node 202. This may include information about the extent of computing resources being used by any application 210, as well as other such information.

[0080] At block 604, a confirmation may be received from the central assessment system. For example, the availability agent 244 may generate and send a confirmation message in response to the information message received from the first agent 216. In one example, the confirmation message is generated with the same periodicity as the information message.

[0081] At block 606, the agent may monitor any changes in the periodicity of confirmation messages received by the agent. For example, the first agent 216 may monitor any changes in the periodicity of confirmation messages received from the availability agent 244. In the event of an increase in periodicity, the first agent 216 may conclude that the first computing node 202 may be experiencing congestion and thus delays between consecutive confirmation messages are occurring. Therefore, the absence of any confirmation messages may indicate that the first computing node 202 may have become inoperable and is no longer communicating with the evaluation system 242.

[0082] At block 608, the agent may isolate the computing node. For example, the first agent 216 may initiate isolation of the first computing node 202 upon determining that the periodicity of the confirmation messages from the evaluation system 242 has increased or in the event that the confirmation messages have stopped. The first agent 216 may then initiate termination of one of the applications, such as the application 210-1, which is not functioning properly or has experienced a failure. Once execution is terminated, the first agent 216 may cut off any communication between the application 210 and the first network card 214 within the first computing node 202. Once communication with the first network card 214 has ended, the application 210-1 may no longer be able to access any network, storage system (e.g., storage system 222), or any other device.

[0083] At block 610, a failover process may be performed on the affected computing node. For example, once application 210-1 has been isolated and unable to access storage system 222, devices, or other services, availability agent 244 may determine a candidate computing node (such as second computing node 204) and initiate a failover of application 210-1 to second computing node 204.

[0084] Figure 7 A computing environment 700 implementing a non-transitory computer-readable medium for handling application failover within a cluster is illustrated according to an example. In an example implementation, the computing environment 700 may be, for example, the cluster 200. In an example, the computing environment 700 includes a processor(s) 702 communicatively coupled to a non-transitory computer-readable medium 704 via a communication link 706. The processor(s) 702 may have one or more processing resources for fetching and executing computer-readable instructions from the non-transitory computer-readable medium 704. The processor(s) 702 and the non-transitory computer-readable medium 704 may be implemented, for example, in the cluster 200.

[0085] The non-transitory computer readable medium 704 may be, for example, an internal storage device or an external storage device. In one example, the communication link 706 may be a direct communication link, such as any memory read / write interface. In another example, the communication link 706 may be an indirect communication link, such as a network interface. The communication link 706 may be a single network or a combination of multiple networks and may use a variety of different communication protocols.

[0086] The processor(s) 702 and the non-transitory computer-readable medium 704 may also be communicatively coupled to a storage element 708. The storage element 708 enables the device hosting the processor(s) 702 and the non-transitory computer-readable medium 704 to service requests made by client devices. For example, a client device may make a request to the first cluster node 202 to access a file or perform a particular operation. To service the request, the first cluster node 202 may access the storage element and perform operations on the data stored therein.

[0087] refer to Figure 7 In one example, the non-transitory computer readable medium 704 includes (multiple) instructions 710 that cause (multiple) processors 702 to handle a failure in the operation of an application 210 in the cluster 200. For example, the instructions 710 can be executed to cause a first agent 216 running on a first network card 214 to periodically transmit information to an evaluation system 242. The first network card 214 is installed in a first computing node 202. The information message transmitted to the evaluation system 242 includes information related to an application 210 that can be installed and executed on the first computing node 202.

[0088] Instructions 710 may thereafter be executed to monitor for incoming response messages that may have been generated as confirmations of the information being transmitted by the first agent 216. In an example, the incoming confirmation messages may have a periodicity that may be based on the periodicity of the information messages or may be predefined, such as based on the control policy 304. Instructions 710 may be further executed to isolate the first computing node 202 based on receipt of the incoming confirmation messages. For example, if the periodicity of the incoming confirmation messages is greater than the predefined periodicity, a failover operation is performed during which an application that is not functioning properly (e.g., application 210-1) may be moved to a different computing node, such as the second computing node 204.

[0089] Although examples of the present disclosure have been described in language specific to structural features and / or methods, it should be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, specific features and methods are disclosed and described as examples of the present disclosure.

Claims

1. A processing system comprising: at least one processor; as well as A non-transitory machine-readable storage medium comprising instructions executable by the at least one processor to: sending control information from the system to a first programmable network adapter in a first computing node over a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receiving, at the system, the information message from the first agent executing on the first programmable network adapter in response to the control information, wherein the information message is related to a program executing on the first computing node; determining a second periodicity for sending the information message by the first agent to the system via the network; comparing the determined second periodicity with the first periodicity specified in the control information; moving the program from the first computing node to a second computing node based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; as well as Instructions are sent to a second agent executing on a second programmable network adapter in the second computing node to manage the program on the second computing node.

2. The system according to claim 1, wherein: The first agent monitors the program executing on the first computing node based on a control policy, wherein the control policy identifies the program monitored by the first agent and operating parameters of the program, and wherein the information message includes the operating parameters.

3. The system of claim 1 , wherein the instructions are executable by the at least one processor to: Based on determining that the determined second periodicity is greater than the first periodicity specified in the control information, the first agent is instructed to isolate the program from a storage system.

4. The system of claim 1 , wherein the instructions are executable by the at least one processor to: processing an operating parameter related to the procedure and obtained from the information message based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; and comparing the operating parameter to a predetermined value, Wherein moving the program from the first computing node to the second computing node is further based on a comparison of the operating parameter with the predetermined value.

5. The system of claim 1 , wherein the instructions are executable by the at least one processor to: obtaining configuration information for the program, wherein the configuration information defines parameters used by the program during operation of the program; and sending the configuration information from a system to the second agent executing on the second programmable network adapter in the second computing node, The program on the second computing node operates according to the configuration information.

6. The system according to claim 4, wherein: The operational parameters include parameters related to the health of the first computing node.

7. The system according to claim 4, wherein: The operating parameters are specified in the control information sent from the system to the first programmable network adapter.

8. The system of claim 1, wherein the instructions are executable by the at least one processor to identify the second computing node based on computing resources used to process the program.

9. A method for failover, comprising: sending control information from a system to a first programmable network adapter in a first computing node over a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receiving, at the system, the information message from the first agent executing on the first programmable network adapter in response to the control information, wherein the information message is related to a program executing on the first computing node; determining, by the system, a second periodicity for the first agent to send the information message to the system via the network; comparing the determined second periodicity with the first periodicity specified in the control information; moving, by the system, the program from the first computing node to a second computing node as part of a failover process based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; as well as After the failover process, the system sends instructions to a second agent executing on a second programmable network adapter in the second computing node to manage the program on the second computing node.

10. The method according to claim 9, comprising: Based on determining that the determined second periodicity is consistent with the first periodicity specified in the control information, the system determines that the first computing node operates in a high-performance manner.

11. The method according to claim 9, further comprising: Based on determining that the determined second periodicity is greater than the first periodicity specified in the control information, the first agent is instructing, by the system, to isolate the program from a storage system.

12. The method according to claim 9, further comprising: Based on determining that the determined second periodicity is greater than the first periodicity specified in the control information: processing, by the system, operating parameters associated with the program and obtained from the information message, the operating parameters being specified by the control information; and comparing, by the system, the operating parameter with a predetermined value, Wherein moving the program from the first computing node to the second computing node is further based on a comparison of the operating parameter with the predetermined value.

13. The method according to claim 11, further comprising: After instructing the first agent to quarantine the program, obtaining, by the system, configuration information for the program, wherein the configuration information defines parameters used by the program in the first computing node during operation of the program; as well as sending the configuration information from a system to the second agent executing on the second programmable network adapter in the second computing node, The program on the second computing node operates according to the configuration information.

14. The method according to claim 9, further comprising: The second computing node is identified by the system based on computing resources used to process the program.

15. A non-transitory computer readable medium comprising instructions that, when executed, cause a system to: sending control information from the system to a first programmable network adapter in a first computing node over a network, the control information specifying initialization of a first agent at the first programmable network adapter and a first periodicity for the first agent to send information messages; receiving, at the system, the information message from the first agent executing on the first programmable network adapter in response to the control information, wherein the information message is related to a program executing on the first computing node; determining a second periodicity for sending the information message by the first agent to the system via the network; comparing the determined second periodicity with the first periodicity specified in the control information; based on determining that the determined second periodicity is greater than the first periodicity specified in the control information, moving the program from the first computing node to a second computing node as part of a failover process; as well as Instructions are sent to a second agent executing on a second programmable network adapter in a second computing node to manage the program on the second computing node.

16. The non-transitory computer readable medium of claim 15, wherein the instructions, when executed, cause the system to: Based on determining that the determined second periodicity is greater than the first periodicity specified in the control information, the first agent is instructed to isolate the program from a storage system.

17. The non-transitory computer readable medium of claim 15, wherein the instructions, when executed, cause the system to: processing an operating parameter related to the procedure and obtained from the information message based on determining that the determined second periodicity is greater than the first periodicity specified in the control information; and comparing the operating parameter to a predetermined value, Wherein moving the program from the first computing node to the second computing node is further based on comparing the operating parameter to the predetermined value.

18. The non-transitory computer readable medium of claim 16, wherein the instructions, when executed, cause the system to: After instructing the first agent to quarantine the program, obtaining configuration information for the program, wherein the configuration information defines parameters used by the program in the first computing node during operation of the program; and sending the configuration information from a system to the second agent executing on the second programmable network adapter in the second computing node, The program on the second computing node operates according to the configuration information.

19. The non-transitory computer readable medium of claim 17, wherein: The operating parameters include parameters related to the health of the first computing node and are specified in the control information sent from the system to the first programmable network adapter.

20. The non-transitory computer readable medium of claim 15, wherein the instructions, when executed, cause the system to: identify the second computing node based on computing resources used to process the program.

Images