Isolation of non-responsive ports in the network fabric

Abstract

A computer-implemented method according to one aspect includes determining whether an operating system of a node in a distributed computing environment is functioning properly by sending a first management query to the node; determining whether the node has an active communication link by sending a second management query to a port associated with the node in response to determining that the operating system of the node is not functioning properly; and resetting the node's active communication link by sending a reset request to the port associated with the node in response to determining that the node has an active communication link.

Description

[Technical field]

[0001] The present invention relates to distributed computing systems, and more particularly, the present invention relates to identifying and addressing non-responsive ports of nodes in a distributed computing system. [Background technology]

[0002] Modern distributed computing environments utilize efficient sharing of computing and storage resources to execute computing tasks. These environments utilize high speed, low latency communication mediums to distribute data across multiple interconnected nodes for processing. Problems arise, however, when one or more nodes malfunction and yet still receive incoming data for processing. This can result in a situation where the entire distributed computing environment needs to be power cycled in order for processing to continue in an optimized manner. Traditional means for identifying and removing failed nodes are unable to accurately identify and respond to malfunctioning nodes that have active communication ports.

[0003] More specifically, modern high-speed interconnect solutions support high-speed data communication. At such speeds, ports that are not experiencing physical link issues and are able to negotiate the correct speed without errors will quickly cause congestion in the network fabric and application flows when they are unable to consume data at the same speed. For example, a Host Channel Adapter (HCA) may have trouble transferring data to host memory because of hardware issues with its Peripheral Component Interconnect (PCI) transfers or because its Direct Memory Access (DMA) hardware is malfunctioning.

[0004] A similar problem can occur if an entire node malfunctions and the system firmware preempts the execution of the operating system while it is processing the hardware fault. If the fault handler takes a significant amount of time to execute, the network adapter queues fill up quickly and become unable to receive new packets. On the other hand, if a local or remote HCA malfunction prevents an application from sending or receiving data with the required integrity and speed, the application may not be able to complete transactions.

[0005] If the entire fabric malfunctions due to a crashed node with active links, the situation becomes extremely critical. In many real-world failure scenarios, an interconnect link can go down quickly if it cannot run the operating system. Furthermore, many interconnect fabrics are highly reliable networks that use hardware credit flow control mechanisms and have timeouts for head-of-queue packets and all packets in the transmit queue. Forwarding standards address components that are present in the network but cannot send or receive packets.

[0006] The standard is defined so that the presence of a malfunctioning component (e.g., a component that is not releasing credits because it cannot receive data) will not block the entire fabric for such a long period of time that it will collapse. However, if a piece of software is in a critical section and must repeatedly communicate with the failed connection in order to leave the critical section until an error occurs or it succeeds, other pieces of software (such as cluster grid logic) may not be able to communicate as all send port credits are repeatedly consumed by the failed component. Thus, the software cannot provide I / O service (because it is in the critical section) nor can it abort the critical section, since the communication associated with the critical section would detect the failure and block other traffic that could communicate the failure to the entire grid.

[0007] Furthermore, real-world test scenarios show that network equipment often fails to properly implement forwarding standards that release transmission credits consumed by non-responsive nodes in the interconnect fabric. Healthy network adapters and switches exhibit erratic behavior in the presence of unhealthy nodes in the fabric, and can even lead to a situation where the entire fabric freezes until the unhealthy element is physically removed from the fabric. In some cases, failures can persist even after removal of the unhealthy element, requiring restarts or resets of elements (switches and nodes) in the fabric, potentially resulting in increased downtime.

[0008] Therefore, timely detection of unresponsive nodes becomes a critical task that can prevent the catastrophic effects of their operation interruption. Summary of the Invention

[0009] A computer-implemented method according to one aspect includes determining whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node, determining whether the node has an active communication link by sending a second management query to a port associated with the node in response to determining that the node has an active communication link by sending a reset request to the port associated with the node, and resetting the active communication link of the node by sending a reset request to the port associated with the node in response to determining that the node has an active communication link.

[0010] In this manner, as a result of the distributed monitoring described above, problematic nodes having faulty operating system functionality or faulty network adapters and active links may be fenced (e.g., identified and removed from the distributed computing environment), which may eliminate communication backlogs caused by these problematic nodes in the distributed computing environment, and thus may improve the functionality of all nodes in the distributed computing environment.

[0011] In one aspect, both the first management query and the second management query are high priority queries. In this manner, data communication issues within the distributed computing environment may be minimized by prioritizing distributed monitoring of nodes within the environment.

[0012] According to another aspect, a computer program product for isolating a non-responsive port in a network fabric includes a computer readable storage medium having program instructions embodied thereon, the computer readable storage medium being not a transient signal itself, the program instructions being executable by a processor to cause the processor to perform a method including: utilizing the processor to determine whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node; in response to determining that the operating system of the node is not functioning properly, utilizing the processor to determine whether the node has an active communications link by sending a second management query to a port associated with the node; and in response to determining that the node has an active communications link, utilizing the processor to reset the active communications link of the node by sending a reset request to the port associated with the node.

[0013] In one aspect, a monitor node in a distributed computing environment is only permitted to send a predetermined number of reset requests to another node within a predetermined period of time.

[0014] In this way, a faulty monitoring node can be prevented from disrupting other functioning nodes with unwarranted reset requests.

[0015] According to another aspect, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor, the logic configured to determine whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node, determine whether the node has an active communications link by sending a second management query to a port associated with the node in response to determining that the node has an active communications link, and reset the active communications link of the node by sending a reset request to the port associated with the node in response to determining that the node has an active communications link.

[0016] According to another aspect, a computer-implemented method includes transmitting a first management query from a first node of a distributed computing environment to a second node of the distributed computing environment, the first management query being associated with an operating system of the second node; in response to a failure at the first node to receive a response to the first management query from the second node, transmitting a second management query from the first node to one or more communications ports associated with the second node; and in response at the first node to receiving a response from the one or more communications ports associated with the second node indicating that the second node has an active communications link, transmitting, by the first node, a link reset request to one or more communications ports associated with the second node requesting a reset of one or more communications ports of the second node having an active communications link.

[0017] In this manner, the first node may act as a management node and, in response to determining a particular scenario in which the second node has an operating system problem or a network adapter problem while also having an active communication link, may trigger the removal of the second node from the distributed computing environment, which may eliminate network congestion in the distributed computing environment caused by the second node, thereby improving performance of the first node and all other computing nodes in the distributed computing environment.

[0018] According to another aspect, a computer program product for implementing a management node in a distributed computing environment includes a computer readable storage medium having program instructions embodied thereon, the computer readable storage medium being not a transient signal itself, the program instructions being executable by the processor to cause the processor to perform a method including: utilizing the processor to transmit, from a first node of the distributed computing environment to a second node of the distributed computing environment, the first management query being associated with an operating system of the second node; in response to a failure at the first node to receive a response to the first management query from the second node, utilizing the processor to transmit, from the first node to one or more communications ports associated with the second node, a second management query; and in response at the first node to receiving, from the one or more communications ports associated with the second node, a response to the second management query indicating that the second node has an active communications link, utilizing the processor to transmit, by the first node, a link reset request to one or more communications ports associated with the second node requesting that one or more communications ports of the second node having an active communications link be reset.

[0019] Other aspects and embodiments of the invention will become apparent from the following detailed description, taken in conjunction with the drawings, illustrating by way of example the principles of the invention. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 illustrates a cloud computing environment according to one embodiment of the present invention. [Diagram 2] FIG. 2 illustrates an abstraction model layer according to an aspect of the present invention. [Diagram 3] FIG. 2 illustrates a cloud computing node according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates a hierarchical data storage system according to one embodiment of the present invention. [Diagram 5] 1 is a flowchart of a method for isolating a non-responsive port in a network fabric according to an aspect of the present invention. [Figure 6] 4 is a flowchart of a method for implementing a management node in a distributed computing environment according to an aspect of the present invention. [Figure 7] FIG. 1 illustrates an exemplary distributed computing network in accordance with an embodiment of the present invention. [Figure 8] 4 is a flowchart of a method for performing node monitoring in a distributed computing environment, according to an aspect of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The following description is made for the purpose of illustrating the general principles of the present invention and is not meant to limit the inventive concepts claimed herein. Moreover, particular features described herein can be used in combination with other described features, in each of the various possible combinations and permutations.

[0022] Unless otherwise expressly defined in this specification, all terms should be given the broadest possible interpretation, including the meaning implied from this specification, as well as the meaning understood by a person of ordinary skill in the art and / or the meaning defined in dictionaries, treatises, etc.

[0023] It should also be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless otherwise indicated. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0024] The following description discloses several aspects of isolating non-responsive ports in a network fabric.

[0025] In one general aspect, a computer-implemented method includes determining whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node; in response to determining that the operating system of the node is not functioning properly, determining whether the node has an active communications link by sending a second management query to a port associated with the node; and in response to determining that the node has an active communications link, resetting the active communications link of the node by sending a reset request to the port associated with the node.

[0026] In this manner, as a result of the distributed monitoring described above, problematic nodes having faulty operating system functionality or faulty network adapters and active links may be isolated (e.g., identified and removed from the distributed computing environment), which may eliminate communication backlogs caused by these problematic nodes in the distributed computing environment, and thus may improve functionality of all nodes in the distributed computing environment.

[0027] In one aspect, both the first management query and the second management query are high priority queries. In this manner, data communication issues within the distributed computing environment may be minimized by prioritizing distributed monitoring of nodes within the environment.

[0028] In another general aspect, a computer program product for isolating a non-responsive port in a network fabric includes a computer readable storage medium having program instructions embodied thereon, the computer readable storage medium being not a transient signal itself, the program instructions being executable by a processor to cause the processor to perform a method including: utilizing the processor to determine whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node; in response to determining that the operating system of the node is not functioning properly, utilizing the processor to determine whether the node has an active communications link by sending a second management query to a port associated with the node; and in response to determining that the node has an active communications link, utilizing the processor to reset the active communications link of the node by sending a reset request to the port associated with the node.

[0029] In one aspect, a monitor node in a distributed computing environment is only permitted to send a predetermined number of reset requests to another node within a predetermined period of time.

[0030] In this way, a faulty monitoring node can be prevented from disrupting other functioning nodes with unwarranted reset requests.

[0031] In another general aspect, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor, the logic being configured to determine whether an operating system of a node of a distributed computing environment is functioning properly by sending a first management query to the node, determine whether the node has an active communications link by sending a second management query to a port associated with the node in response to determining that the node has an active communications link, and reset the active communications link of the node by sending a reset request to the port associated with the node in response to determining that the node has an active communications link.

[0032] In another general aspect, a computer-implemented method includes transmitting a first management query from a first node of a distributed computing environment to a second node of the distributed computing environment, the first management query being associated with an operating system of the second node; in response to a failure at the first node to receive a response to the first management query from the second node, transmitting a second management query from the first node to one or more communications ports associated with the second node; and in response at the first node to receiving a response to the second management query from the one or more communications ports associated with the second node indicating that the second node has an active communications link, transmitting, by the first node, a link reset request to one or more communications ports associated with the second node requesting a reset of one or more communications ports of the second node having an active communications link.

[0033] In this manner, the first node may act as a management node and, in response to determining a particular scenario in which the second node has an operating system problem or a network adapter problem while also having an active communication link, may trigger the removal of the second node from the distributed computing environment, which may eliminate network congestion in the distributed computing environment caused by the second node, thereby improving performance of the first node and all other computing nodes in the distributed computing environment.

[0034] In another general aspect, a computer program product for implementing a management node within a distributed computing environment includes a computer readable storage medium having program instructions embodied thereon, the computer readable storage medium being not a transitory signal itself, the program instructions including: utilizing the processor to transmit a first management query from a first node of the distributed computing environment to a second node of the distributed computing environment, the first management query being associated with an operating system of the second node; and in response to failure to receive a response to the first management query at the first node from the second node, The method is executable by the processor to cause the processor to perform a method including: utilizing the processor to transmit a second management query from the first node to one or more communication ports associated with the second node; and in response to receiving, at the first node, a response to the second management query from the one or more communication ports associated with the second node indicating that the second node has an active communication link, utilizing the processor to transmit a link reset request by the first node to one or more communication ports associated with the second node requesting that the one or more communication ports of the second node having the active communication link be reset.

[0035] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, aspects of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0036] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and published with minimal administrative effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0037] Its features are as follows:

[0038] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0039] Pervasive Network Access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0040] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated on demand. There is location independence in that consumers generally have no control or knowledge of the exact location of the provided resources, although it may be possible to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0041] Rapid Elasticity: Capabilities can be rapidly and elastically provisioned, sometimes automatically, to rapidly scale out and rapidly release to rapidly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

[0042] Service Metering: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a certain level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0043] The service model is as follows:

[0044] Software as a Service (SaaS): The capability offered to a consumer to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). With the possible exception of limited user-specific application configuration settings, the consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application features.

[0045] Platform as a Service (PaaS): The capability offered to a consumer to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, such as networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0046] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0047] The deployment model is as follows:

[0048] Private Cloud: The cloud infrastructure is operated exclusively for an organization. This cloud infrastructure can be managed by the organization or a third party and can exist on-premise or off-premise.

[0049] Community Cloud: Cloud infrastructure is shared across multiple organizations to support a specific community of shared concerns (e.g., mission, security requirements, policies, and compliance considerations). This cloud infrastructure may be managed by the organizations or a third party and may exist on-premise or off-premise.

[0050] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0051] Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0052] A cloud computing environment is a service-driven approach that focuses on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0053] Referring now to FIG. 1, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, can communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described herein above (not shown). This enables the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on a local computing device. It should be understood that the types of computing devices 54A-54N shown in FIG. 1 are intended as examples only, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connections (e.g., using a web browser).

[0054] Referring now to Figure 2, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 1) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 2 are intended to be exemplary only, and that aspects of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0055] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0056] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0057] In one example, the management layer 80 may provide the functions described below: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and charging or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-positioning and procurement of cloud computing resources anticipated to be required in the future according to SLAs.

[0058] The workload tier 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and distributed computing 96.

[0059] 3, a schematic diagram of an example cloud computing node is shown. Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the aspects of the invention described herein. Cloud computing node 10 may nevertheless be implemented and / or perform any of the functionality described herein above.

[0060] Cloud computing node 10 includes computer system / server 12 that is operable with many other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0061] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0062] 3, computer system / server 12 in cloud computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 coupling various system components including system memory 28 to processor 16.

[0063] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0064] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0065] The system memory 28 may include computer system readable media in the form of volatile memory such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such a case, each may be connected to the bus 18 by one or more data media interfaces. As further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of aspects of the present invention.

[0066] A program / utility 40 having a set (at least one) of program modules 42 may be stored within memory 28, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methodologies of aspects of the present invention as described herein.

[0067] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur through an input / output (I / O) interface 22. Additionally, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 through a bus 18. Although not shown, it is understood that other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0068] Referring now to FIG. 4, a storage system 400 according to one embodiment is shown. It should be noted that some of the elements shown in FIG. 4 may be implemented as hardware and / or software according to various embodiments. The storage system 400 may include a storage system manager 412 for communicating with a plurality of media on at least one upper storage tier 402 and at least one lower storage tier 406. The upper storage tier 402 may preferably include one or more random access and / or direct access media 404, such as hard disks in a hard disk drive (HDD), non-volatile memory (NVM), solid state memory in a solid state drive (SSD), flash memory, SSD arrays, flash memory arrays, etc., or others described herein or known in the art, or combinations thereof. The lower storage tier 406 may preferably include one or more lower performance storage media 408, including sequential access media such as magnetic tape in a tape drive and / or optical media, slow access HDDs, slow access SSDs, etc., or others described herein or known in the art, or combinations thereof. The one or more additional storage tiers 416 may include any combination of storage memory media as desired by the designer of the system 400. Also, either the upper storage tier 402 and / or the lower storage tier 406 may include any combination of storage devices and / or storage media.

[0069] The storage system manager 412 may communicate with the storage media 404, 408 on the upper storage tier 402 and the lower storage tier 406 via a network 410, such as a storage area network (SAN) or some other suitable network type, as shown in FIG. 4. The storage system manager 412 may also communicate with one or more host systems (not shown) via a host interface 414, which may or may not be part of the storage system manager 412. The storage system manager 412 and / or any other components of the storage system 400 may be implemented in hardware and / or software and may utilize a processor (not shown) for executing commands of a type known in the art, such as a central processing unit (CPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc. Of course, any configuration of storage system may be used, as would be apparent to one of ordinary skill in the art upon reading this description.

[0070] In more embodiments, storage system 400 may include any number of data storage tiers and may include the same or different storage memory media within each storage tier. For example, each data storage tier may include the same type of storage memory media, such as HDDs, SSDs, sequential access media (tapes in tape drives, optical disks in optical disk drives, etc.), direct access media (CD-ROMs, DVD-ROMs, etc.), or any combination of media storage types. In one such configuration, upper storage tier 402 may include a majority of SSD storage media for storing data in a higher performance storage environment, while the remaining storage tiers, including lower storage tier 406 and additional storage tier 416, may include any combination of SSDs, HDDs, tape drives, etc. for storing data in a lower performance storage environment. In this manner, data that is accessed more frequently, data that has a higher priority, data that needs to be accessed more quickly, etc. may be stored in upper storage tier 402, while data that does not have one of these attributes may be stored in additional storage tiers 416, including lower storage tier 406. Of course, those of ordinary skill in the art, upon reading this description, will be able to devise many other combinations of storage media types for implementing different storage schemes in accordance with aspects presented herein.

[0071] According to some aspects, a storage system (e.g., 400) may include logic configured to receive a request to open a data set, logic configured to determine whether the requested data set is stored in multiple relevant portions in a lower storage tier 406 of the hierarchical data storage system 400, logic configured to move each relevant portion of the requested data set to a higher storage tier 402 of the hierarchical data storage system 400, and logic configured to assemble the requested data set on the higher storage tier 402 of the hierarchical data storage system 400 from the relevant portions.

[0072] Of course, this logic may be implemented in a variety of ways, as a method on any device and / or system, or as a computer program product.

[0073] Referring now to Figure 5, there is shown a flow chart of a method 500, according to one embodiment. Method 500 may be performed in accordance with the present invention in various embodiments, particularly in any of the environments shown in Figures 1-4 and 7. Of course, method 500 may include more or less operations than those specifically depicted in Figure 5, as will be appreciated by those of skill in the art upon reading this specification.

[0074] The steps of method 500 may be performed by any suitable component of an operating environment. For example, in various embodiments, method 500 may be performed in part or in whole by one or more servers, computers, or any other device having one or more processors therein. A processor, e.g., a processing circuit, chip, or module, or a combination thereof, implemented in hardware or software or both, preferably having at least one hardware component, may be utilized in any device to perform one or more steps of method 500. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0075] 5, method 500 may begin at operation 502, where an operating system of a node of a distributed computing environment is determined to be functioning properly by sending a first management query to the node. In one aspect, a distributed computing environment may include multiple nodes (e.g., computing nodes, etc.) that function together to perform one or more computational actions (e.g., running an application, processing data, etc.). For example, a node may include a computing device (e.g., a computer, a server, etc.) having physical, tangible hardware storage and computing resources.

[0076] In one aspect, it may be determined whether the node's network adapters are functioning properly. For example, the first management query may determine whether the node's network adapters are reachable, available, operational, etc. In another aspect, the first management query may verify whether the node's operating system is reachable (e.g., whether the OS can function using its ports).

[0077] Additionally, in one aspect, a distributed computing environment may implement a variety of applications (e.g., stock market applications, storage services, database services, etc.) In another aspect, the nodes in a distributed computing environment may all be located in the same physical location and connected utilizing point-to-point connections.

[0078] Additionally, in one aspect, the nodes in a distributed computing environment may be located in different physical locations and connected utilizing high speed connections (e.g., Fibre Channel connections, etc.). In another aspect, all the nodes in the distributed computing environment may communicate with one another with low latency over one or more high data rate connections (e.g., Fast Ethernet connections, InfiniBand (IB) connections, hardware level connections, etc.). In yet another aspect, the nodes in the distributed computing environment may share resources (e.g., processing power, memory, etc.) with one another within the distributed computing environment.

[0079] Additionally, in one aspect, the distributed computing environment may include a grid architecture, a network fabric, or the like. In another aspect, a first node of the distributed computing environment may send the first management query directly to a second node (e.g., via a point-to-point connection between the two nodes). In yet another aspect, a first node of the distributed computing environment may send the first management query indirectly to a second node (e.g., via a switch disposed between the two nodes, or the like).

[0080] Also, in one aspect, the first node and the second node may be adjacent nodes in a distributed computing environment. In another aspect, the first node may send a first management query to the second node in response to determining that the second node is an adjacent node of the first node. In yet another aspect, a daemon (e.g., an application or code such as a monitoring agent) running on the first node may send the first management query from the first node to the second node.

[0081] Further, in one aspect, the first management query may include a fabric management query that is terminated within the receiving node's operating system. For example, in an InfiniBand fabric, the first management query may include an SMP (Subnet Management Protocol) MAD (Management Datagram) query of attribute type SMINFO (Subnet Manager Information) that is terminated by the operating system. In another aspect, the receiving node's operating system may transmit a response to the first management query in response to receiving the first management query. For example, the response may be transmitted only if the receiving node's operating system is operating correctly. In another example, the response may not be transmitted if the receiving node's operating system is not operating correctly, not functioning, etc.

[0082] Further, method 500 may proceed to operation 504, where in response to determining that the operating system of the node is not functioning properly, determining whether the node has an active communication link by sending a second management query to a port associated with the node. In one aspect, in response to determining that a response to the first management query is not received from the node within a predetermined period of time, it may be determined that the operating system of the node is not functioning properly.

[0083] For example, in response to determining that a response to the first management query is not received from the node within a predetermined time period, the first management query may be retransmitted to the node one or more times. In another example, in response to determining that a response is not received from the node after retransmitting the first management query to the node a predetermined number of times, it may be determined that the node is not functioning properly.

[0084] Additionally, in one aspect, in response to determining that a response to the first management query is received from the node within a predetermined time period, it may be determined that the operating system of the second node is functioning properly. For example, in response to determining that the node's operating system is functioning properly, the first management query may be sent again to the node at a later time according to a predetermined monitoring schedule.

[0085] Also, in one aspect, the response may be received at the node that sent the first management query. For example, if a first node in a distributed computing environment sends a first management query to a second node, the response from the second node may be received at the first node. In another aspect, in response to determining that a response to the first management query is not received from the node within a predetermined period of time, a second management query may be sent to a port associated with the node. Additionally, the second management query may ascertain whether the node has an active communication link by querying an associated network port whether that link is active. In one aspect, an active communication link may include an active connection between a port on the node and another port on another node. In this case, the first node of the distributed computing environment may send the second management query directly to its local port connected to the second node. In another aspect, an active communication link may include an active connection between a port on the node and another port on a switch. In this case, the first node of the distributed computing environment may send the second management query to the switch to query the state of the port of the second node. In yet another aspect, the first node of the distributed computing environment may send the second management query indirectly (e.g., via a switch disposed between the two nodes or using a direct link) or directly to a port in the second node. For example, in response to determining that the operating system (OS) of the second node is not functioning properly, the first node may examine all ports associated with the second node to determine whether any ports of the second node are present and active.

[0086] Further, in one aspect, a second node may be determined to have an active communication link if a port associated with the second node is present and active. In another aspect, the queried network port may respond to the second management query by indicating any active links maintained by the node (e.g., via one or more ports of the node, etc.). In yet another aspect, the response may be received at the node that sent the second management query. For example, if a first node of a distributed computing environment sends a second management query to a port associated with a second node, a response from the queried network port may be received at the first node.

[0087] Further, method 500 may proceed to operation 506, where in response to determining that the node has an active communication link, an active communication link of the node is reset by transmitting a reset request to one or more network ports associated with the node. In one aspect, the ports associated with the node may include one or more ports connected to the node. For example, in response to identifying an active port in the node in response to a second management query, the node may be determined to have an active communication link. In another aspect, in response to failing to identify an active port in the node in response to a second management query, the node may be determined to not have an active communication link. For example, in response to determining that the node does not have an active communication link, the first management query may be transmitted again to the node at a later time according to a predetermined monitoring schedule.

[0088] Also, in one aspect, in response to determining that the operating system of the node is not functioning properly and at the same time the node has an active communications link, the node may be immediately flagged as having a problem, for example, by changing metadata identifying the node in another node performing monitoring (e.g., by sending an administrative query, etc.).

[0089] Further, in one aspect, in response to determining that the operating system of the node is not functioning properly while the node has an active communications link, a predetermined count may be incremented and monitoring of the node may be increased before flagging the node as problematic. For example, a first administrative query may be repeatedly sent to the node as part of scheduled periodic monitoring activity at a first predetermined frequency.

[0090] Further, in one example, in response to determining that the predetermined count has been incremented, the first predetermined frequency of the scheduled periodic monitoring activity may be changed to a second predetermined frequency that is greater than the first predetermined frequency. In another example, the first management query may then be retransmitted to the node according to the second predetermined frequency. In this manner, when the predetermined count is first incremented (indicating that the node's operating system is not functioning properly and at the same time the node has an active communications link), follow-up management queries may be transmitted to the node at a high frequency to quickly determine that there is a problem with the node.

[0091] Further, in one example, a predetermined count may be incremented each time it is determined that the node's operating system is not functioning properly while at the same time the node has an active communication link. In another example, when the predetermined count exceeds a threshold, then the node may be flagged as having a problem. In yet another example, upon determining that the node is not having a problem (e.g., upon determining that the node's operating system is functioning properly or that the node does not have an active communication link), the predetermined count may be reset. In this manner, false positive identification of problematic nodes may be minimized.

[0092] Additionally, in one aspect, in response to flagging the node as having a problem, a reset request may be sent to one or more network ports associated with the node to reset the node's active communication links. In another aspect, a reset request may be sent to the node ports to reset all active links in the node. In either case, a running operating system and responding network ports may need to rejoin the fabric, which may result in the node having no active links since either one or the other is not functioning.

[0093] In another example, in the case of a false positive, the reset request may not remove the node from the distributed computing environment for a significant amount of time (e.g., a link reset would result in an active node) because the node's operating system and network ports are functional and the node will rejoin the network after the link reset action is completed.

[0094] Also, in one aspect, the entity sending the reset request may log the date and time of the reset request (eg, using a timestamp, etc.).

[0095] Further, in one aspect, one or more managers in the distributed computing environment may then identify the lack of an active link in the node and remove the node from the distributed computing environment. For example, the one or more managers may include a grid manager, a fabric manager, etc. In another example, the node may then be reset (e.g., by one or more managers) and thereafter rejoin the distributed computing environment when its network ports and operating system are functional. The use of a link reset action (as opposed to a link shutdown, which would completely physically isolate the node from the network ports associated with the node) may allow a functional node to automatically rejoin the fabric, thereby minimizing both the impact of a false positive detection and the need for manual intervention to resume I / O operations.

[0096] Further, in one aspect, each node in the distributed computing environment may be designated as a monitoring node. For example, each monitoring node may send a first management query and a second management query (if necessary) to all neighboring nodes using the methods described herein. In another example, if any of the neighboring nodes are identified as problematic, the monitoring node may reset all active links in the problematic node. In yet another example, the problematic node may then be removed from the distributed computing environment and reset by one or more managers.

[0097] In this manner, monitoring within a distributed computing environment may be distributed across all nodes within the environment, thereby mitigating the impact of one or more node failures within the environment.

[0098] Furthermore, in one aspect, a monitoring node in a distributed computing environment may only be permitted to send a predetermined number of reset requests to another node within a predetermined time period. In another aspect, once the number of reset requests sent by a monitoring node reaches a predetermined threshold, the monitoring node may continue to perform monitoring actions but may not be able to send additional reset requests (e.g., until a predetermined time has elapsed). In this manner, a faulty monitoring node may be prevented from disrupting another functioning node with an unwarranted reset request.

[0099] Also, in one aspect, both the first and second management queries may be high priority in-band management queries or may use high SL (service level) or high VL (virtual lane) levels. In this manner, data communication issues within the distributed computing environment may be minimized by prioritizing distributed monitoring of nodes within the environment.

[0100] In yet another aspect, the first management query and the second management query may use a direct routed packet format (wherein the routing hops between the sending port and the receiving port are recorded in the packet), thus minimizing data communication problems in the distributed computing environment that impact the ability to route traffic.

[0101] As a result of the above-mentioned distributed monitoring, problematic nodes having faulty operating system functionality or faulty network adapters and active links may be isolated (e.g., identified and removed from the distributed computing environment), which may eliminate communication backlogs caused by these problematic nodes in the distributed computing environment, thus improving the functionality of all nodes in the distributed computing environment.

[0102] Referring now to Figure 6, a flow chart of a method 600 for implementing an administrative node in a distributed computing environment is shown, according to one embodiment. Method 600 may be performed in accordance with the present invention in various embodiments, particularly in any of the environments shown in Figures 1-4 and 7. Of course, method 600 may include more or less operations than those specifically depicted in Figure 6, as will be appreciated by those of skill in the art upon reading this specification.

[0103] The steps of method 600 may be performed by any suitable component of an operating environment. For example, in various embodiments, method 600 may be performed in part or in whole by one or more servers, computers, or any other device having one or more processors therein. A processor, e.g., a processing circuit, chip, or module, or a combination thereof, implemented in hardware or software or both, preferably having at least one hardware component, may be utilized in any device to perform one or more steps of method 600. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0104] 6, method 600 may begin at operation 602, where a first management query is sent from a first node of a distributed computing environment to a second node of the distributed computing environment, where the first management query is associated with an operating system of the second node. Further, method 600 may proceed to operation 604, where in response to failing to receive a response to the first management query from the second node at the first node, a second management query is sent from the first node to one or more communication ports associated with the second node.

[0105] Further, method 600 may proceed to operation 606, where, in response to receiving, at the first node, a response to the second management query from one or more communication ports associated with the second node indicating that the second node has an active communication link, the first node sends a link reset request to one or more communication ports associated with the second node requesting that the one or more communication ports of the second node having the active communication link be reset.

[0106] In this manner, the first node may act as a management node and, in response to determining a particular scenario in which the second node has an operating system problem or a network adapter problem while also having an active communication link, may trigger the removal of the second node from the distributed computing environment, which may eliminate network congestion in the distributed computing environment caused by the second node, thereby improving performance of the first node and all other computing nodes in the distributed computing environment.

[0107] In one embodiment, a plurality of additional nodes may be included along with the first node and the second node in the distributed computing environment, hi another embodiment, each node in the distributed computing environment may utilize a first management query and a second management query to monitor neighboring nodes.

[0108] 7 illustrates an exemplary distributed computing network 700, as shown in one exemplary embodiment. As shown, the distributed computing network 700 includes a plurality of nodes 702A-E. In one embodiment, the network 700 may include more or less than the plurality of nodes 702A-E.

[0109] Further, in one aspect, the multiple nodes 702A-E are interconnected via network connections 704A-G, 708A-B. In one aspect, the network connections 704A-G, 708A-B may include high speed, low latency data connections (e.g., Fast Ethernet connections, InfiniBand connections, hardware level connections, etc.).

[0110] Additionally, the plurality of communication switches 706A-B enable communication between a first portion of the nodes 702A and 702B and a second node 702C. For example, the communication switches 706A-B may relay data between the first portion of the nodes 702A and 702B and the second node 702C. The communication switches 706A-B also communicate via cross-switch network connections 708A-B to facilitate communication / monitoring between the first portion of the nodes 702A and 702B and the second node 702C.

[0111] Additionally, in one aspect, each of the plurality of nodes 702A-E may function as a monitoring node within the distributed computing network 700 and may communicate with its neighboring nodes to determine the status of such nodes. For example, monitoring node 702C may communicate directly with its first neighboring node 702D to determine the status of that neighboring node 702D. Additionally, monitoring node 702C may communicate with additional neighboring nodes 702A and 702B via communication switches 706A-B to determine their status.

[0112] For example, the monitoring node 702C may send a first management query directly to its neighboring node 702D, where the first management query is associated with the operating system of the receiving node. After failing to receive a response to the first management query from the neighboring node 702D, the monitoring node 702C may send a second management query to a port associated with the neighboring node 702D.

[0113] Also, in response to receiving a response from the port associated with the neighboring node 702D indicating that the neighboring node 702D has an active communication link at the monitoring node 702C, the monitoring node 702C may send a link reset request to the port associated with the second node to reset the port of the neighboring node 702D having the active communication link. In response to receiving the link reset request from the monitoring node 702C, the port associated with the neighboring node 702D may reset all active links in the neighboring node 702D. The grid manager or fabric manager of the distributed computing network 700 may then remove the neighboring node 702D from the distributed computing network 700.

[0114] The monitoring node 702A may also indirectly send a first management query to its additional neighboring nodes 702B and 702C via the communication switches 706A-B. In response to receiving a response to the first management query by its additional neighboring nodes 702B and 702C via any of the communication switches 706A-B, the monitoring node 702A may take no further action until its monitoring schedule indicates that another first management query is to be sent again to its additional neighboring nodes 702B and 702C. For example, the monitoring node 702A may query the neighboring node 702B using a first management query via the network connections 704A-B. Upon receiving a response to the first management query, no further action may be taken by the monitoring node 702A until the monitoring schedule resumes monitoring the monitoring node 702A.

[0115] Upon identifying failure to receive a response to the first management query, a second management query may be sent using network connection 704A to determine whether network link 704B is active. Additionally, the monitoring node 702A repeats the first management query using network connections 704F-G, and if no response is received, repeats the second management query via network connection 704F to determine whether link 704G is active. If both probes using the first management query (via network connections 704A-B and 704F-G) fail and at least one of the network connections (704B or 704G) is found to be active using the second management query, node 702B will be flagged as having a problem. If node 702B is detected as having a problem, it may send a reset request using network connections 704A and 704F to reset network connections 704B and 704G.

[0116] In this manner, each of the plurality of nodes 702A-E may act as a watchdog node within the distributed computing network 700 and may trigger the removal of any problematic nodes within the distributed computing network 700, thereby improving the performance of the remaining nodes within the distributed computing network 700.

[0117] Referring now to Figure 8, a flow chart of a method 800 for performing node monitoring within a distributed computing environment is shown, according to one embodiment. Method 800 may be performed in accordance with the present invention in various embodiments, particularly in any of the environments shown in Figures 1-4 and 7. Of course, method 800 may include more or less operations than those specifically depicted in Figure 8, as will be appreciated by those of skill in the art upon reading this specification.

[0118] The steps of method 800 may be performed by any suitable component of an operating environment. For example, in various embodiments, method 800 may be performed in part or in whole by one or more servers, computers, or any other device having one or more processors therein. A processor, e.g., a processing circuit, chip, or module, or a combination thereof, implemented in hardware or software or both, preferably having at least one hardware component, may be utilized in any device to perform one or more steps of method 800. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0119] As shown in FIG. 8, method 800 may begin at operation 802 where a remote node is probed via a next path. In one aspect, the remote node may include a neighboring node of the monitoring node. Further, method 800 may proceed to decision 804 where a probe status is checked. If, at decision 804, it is determined that a response to the probe has been received by the monitoring module, method 800 may proceed to operation 806 where the next cycle is prepared. For example, the next path to be probed may be updated, a failure count may be set to zero, an operating system (OS) probe status may be saved for the current neighboring node, and a normal delay may be used for the next probe cycle.

[0120] Further, if at decision 804 it is determined that a response to the probe is not received by the monitoring module within a predetermined period of time, method 800 may proceed to decision 808, where the link status is checked for the adjacent node. If at decision 808 it is determined that a probe error exists, method 800 may proceed to operation 806, where the next cycle is prepared. A failure to probe the status of the associated remote port (e.g., a port of a switch) may indicate the presence of another transient (or non-transient) problem in the network fabric that is not handled by this method (which handles failures of endpoint nodes). Thus, monitoring will resume after a normal timeout.

[0121] Further, if at decision 808 it is determined that the link status is up or down, method 800 may proceed to decision 810, where it is determined whether an additional path exists (e.g., a path to an adjacent node of the monitoring node, etc.). If at decision 810 it is determined that an additional path exists, method 800 may proceed to operation 812, where the next path is updated, and then method 800 may proceed to operation 802, where the remote node is probed via the next path.

[0122] Further, if at decision 810 it is determined that no additional paths exist, the method 800 may proceed to decision 814, where it is determined whether an active link is found in the remote node. If at decision 814 it is determined that an active link is not found, the method 800 may proceed to operation 806, where the next cycle is prepared. If at decision 814 it is determined that an active link is found, the method 800 may proceed to operation 816, where a failure count for the remote node is incremented.

[0123] Method 800 may also proceed to decision 818, where it is determined whether the failure count of the remote node has reached a predetermined threshold. If, in decision 818, it is determined that the failure count has reached the threshold, method 800 may proceed to operation 820, where the link in the remote node is reset, and then method 800 may proceed to operation 806, where the next cycle is prepared. In one aspect, a monitoring node performing a link reset may perform a limited number of link resets before the ability to perform link resets is disabled for the monitoring node.

[0124] Further, if in decision 818 it is determined that the failure count has not reached the threshold, the method 800 may proceed to operation 822, where preparation for the next cycle is made, and then the method 800 may proceed to operation 802, where the remote node is probed via the next path. For example, preparing the next cycle may include updating the next path, saving the OS probe status, and using a smaller delay for the next cycle.

[0125] In this way, nodes in the fabric can monitor adjacent nodes in the fabric and trigger the removal of problematic nodes in the fabric, thereby improving the performance of the remaining nodes in the fabric.

[0126] Isolate non-responsive ports in the interconnect fabric

[0127] In one aspect, a mechanism is provided for detecting malfunctioning nodes that appear to be present on the network. Modern enterprise applications often deploy grid architectures where application services are provided by multiple grid nodes interconnected with a high speed interconnect fabric. Fast Ethernet, InfiniBand, or Fibre Channel are exemplary interconnect solutions. Due to the complex requirements of enterprise applications, a variety of protocols are deployed on the interconnect fabric.

[0128] Traditional protocols like Ethernet® or TCP / IP may not be sufficient to meet application requirements for performance, load balancing, and guaranteed data delivery. Applications switch to interconnect protocols that enable hardware level connections and RDMA (Remote Direct Memory Access) transfers. InfiniBand and DCB Ethernet® (Data Center Bridging) fabrics allow reliable peer-to-peer connections implemented at the HCA (Host Channel Adapter) level and using the RDMA protocol.

[0129] The various DCB technologies are also known by the terms Converged Enhanced Ethernet (CEE) or Lossless Ethernet. Grid architecture solutions support both scalability and high availability. The grid can be expanded by adding nodes to the fabric, and if a node fails, the entire grid cluster can shunt the load to the remaining nodes in the grid. Once the issue with the failed node is resolved, the grid can be expanded again to include the previously failed node.

[0130] A management solution is deployed to manage the work distribution and monitor the health status of the grid. Management solutions typically deploy a TCP protocol for communication across either the data fabric or a slow dedicated side network channel. Typically, all nodes communicate with the grid's manager and exchange keep-alive or health messages. If a node reports a failure or is not seen on the management network for a defined period of time, the grid's manager can declare the node as failed. If all nodes in the grid communicate with each other, a TCP protocol using majority voting can also be used to elect a manager.

[0131] A fabric management solution (e.g., a subnet manager) can be deployed on the Grid cluster to discover and configure new nodes in the fabric and handle their disappearance from the network in a timely manner. The fabric manager can perform periodic discovery of the fabric, nodes, and ports and can handle the addition of new ports and the disappearance of existing ports.

[0132] Modern high-speed interconnect solutions support speeds of 100-200 Gigabits per port. At such speeds, ports that are capable of negotiating the correct speed without encountering physical link issues or errors can quickly cause congestion in the network fabric and application flows if they cannot consume data at the same rate. For example, an HCA may have problems transferring data to host memory because it has hardware issues with PCI transfers or because its DMA hardware is malfunctioning.

[0133] A similar problem can occur if an entire node malfunctions and the system firmware preempts the execution of the operating system while it is processing the hardware fault. If the fault handler takes a significant amount of time to execute, the network adapter queues fill up quickly and become unable to receive new packets. On the other hand, if a local or remote HCA malfunction prevents an application from sending or receiving data with the required integrity and speed, the application may not be able to complete transactions.

[0134] Specifically, storage grid solutions need to synchronize certain transactions across the grid cluster when the storage distribution changes (e.g., by adding a new node, removing an existing node, creating or deleting a volume, taking a snapshot, etc.). When this occurs, new storage transactions from clients can continue until the distribution is synchronized across all nodes in the grid. If a node becomes stuck during such a transaction, the manager may need to quickly identify the stuck node and mark it as failed in order to allow client IO to continue.

[0135] If the entire fabric malfunctions due to the presence of a crashed node with active links, the situation becomes extremely critical. The presence of a crashed node with active links in an interconnect fabric is not a well-tested situation. In many real-world failure scenarios, an interconnect link can go down quickly if it cannot run the operating system. Many interconnect fabrics are reliable networks that use hardware credit flow control mechanisms and have timeouts for head-of-queue packets and all packets in the transmit queue. Forwarding standards address components that are present in the network but cannot send or receive packets.

[0136] The standard is defined in such a way that the presence of a malfunctioning component (e.g., one that is not releasing credits because it cannot receive data) does not block the entire fabric for such a long period that it leads to a collapse. For example, in an InfiniBand network, a packet is never stuck in the transmit queue of a switch port for longer than 2-2.5 seconds. Thus, if all switch port credits are consumed by a crashed endpoint, the switch port will be able to provide transmit credits again within that period.

[0137] However, if a piece of software is in a critical section and must communicate with the failed destination multiple times to leave the critical section until an error occurs or it succeeds, other pieces of software (such as cluster grid logic) may not be able to communicate as all send port credits are repeatedly consumed by the failed component. Thus, the software cannot provide I / O service (because it is in the critical section) or abort the critical section, because the communication associated with the critical section would detect the failure and block other traffic that could communicate the failure to the entire grid.

[0138] Furthermore, real-world testing has shown that, despite what may be stated in reliable network assurance and forwarding standards, existing switches and nodes cannot gracefully handle a malfunctioning port in the fabric where a link is present but unable to receive packets. Thus, one misbehaving port in the fabric can bring down an entire grid cluster. Therefore, a method is needed to detect and isolate non-responsive ports in the fabric to allow applications to continue.

[0139] In one aspect, a method is provided for implementing a distributed fabric management solution that can isolate non-responsive ports in an interconnect fabric. Each node in the fabric is responsible for monitoring one or more adjacent ports such that redundant monitoring is achieved. Monitoring is performed using fabric management queries that have high delivery priority and are terminated within the operating system.

[0140] Link presence monitoring is also performed using a high priority fabric management query to the switch (or connected endpoints if there is no switch), which may be terminated in the switch or adapter firmware. A problem within a node is detected when a node port has active links, but none of the active links respond to a management query terminated by the operating system.

[0141] If a problem is reliably detected, the monitoring agent will request the switch to reset the link (or ask for a local link reset in a point-to-point switchless fabric). The fabric management solution is such that activation of the link in data mode is only possible if the node with the discovered link responds at the OS level. Thus, the problematic link will remain inactive for data transfer until the operating system recovers.

[0142] A distributed monitoring solution is provided. Each node monitors two or more neighbors, and redundant monitoring is achieved. The solution implements the SMINFO management discovery attribute, which is terminated by the operating system. Priority transmission of management queries is guaranteed by the interconnection fabric. Each monitoring party performs periodic discovery of its neighbors as follows:

[0143] 1. Neighboring ports are queried using management attributes that terminate in the OS. As soon as at least one neighbor responds, the probe is successful and completes.

[0144] 2. If none of the ports respond, the monitoring node probes all neighboring ports for the presence of a physical link. For queries to switched networks, the switch is used, for point-to-point links, a local port query is used. If at least one port with a link is found, the query stops.

[0145] 3.a) If no ports respond to the SMINFO query and at least one port of the node has a physical link, the node is flagged as a problematic node.

[0146] Under normal operating conditions, a longer monitoring interval is used. If a problem is detected, the monitoring interval is shortened. A timestamp of the problem detection is recorded and reported with the event. After repeated detection of the problem (a configurable number is used), recovery action is taken.

[0147] The recovery action will be to reset the node link with a command to the switch on switched fabrics and to the local port in point-to-point switchless fabrics. A timestamp of the recovery action will be recorded and reported via an event. The success of any monitoring step will result in a reset of the repeated fault detection counter to zero. The timestamp of the switch from a non-zero value to a zero value will also be recorded and reported via an event.

[0148] Monitoring can be enabled or disabled on a per-node basis, and a bullet mechanism can also be enabled. When enabled, a limited number of recovery action bullets are given to each monitor. Once the action bullets are consumed, these bullets are replenished only after a defined period of time has elapsed. This allows a monitor to reset the links of the monitored node no more than a defined number of times during a defined time slice. Once the bullets are consumed, monitoring is not paused and all transition event times are recorded and reported without taking any link reset action.

[0149] The solution is tolerant to node failures (because it is distributed) and data communication collapses (because it uses high priority fabric management traffic).The solution also avoids false positive detections of communication loss (because it uses multiple probes) and false negative detections of communication presence (because it uses both link probes and operating system probes).

[0150] At the same time, the solution is able to detect non-responsive ports and take recovery actions quickly enough to avoid both application failures and fabric collapse. Even if a false positive detection occurs, it is not expected to have a significant impact: once the link is renegotiated, the node with the reset link will join the fabric (since it is able to respond to fabric management commands or begin participating in the fabric) and resume I / O operations.

[0151] InfiniBand port isolation for crashed modules

[0152] In one aspect, in the A9000 grid storage architecture, a module is a building block of the storage grid. A module is a server that has a CPU, memory, peripheral devices, and runs XIV (XIV is a trademark of IBM Corporation) storage software. The XIV storage software is based on the Linux (Linux is a trademark of Linus Torvalds) operating system and XIV storage services, and runs in user space and kernel space.

[0153] In another aspect, a crashed node includes a module that is unable to run the XIV OS and storage services due to a hardware or software malfunction. In another aspect, the A9000 Grid Storage uses InfiniBand interconnects to communicate between modules in a switched or switchless (point-to-point) topology.

[0154] In yet another aspect, port isolation may include shutting down an IB port (eg, on a crashed module) in such a way that data communication with the remainder of the storage grid is not possible.

[0155] In yet another aspect, the IB fabric may include a high-speed InfiniBand network of switches, endpoints, and links that allow endpoints (IB ports on modules) to exchange data. For example, the A9000 system uses 56 Gigabit IB links between modules and has two IB ports per module in a rack configuration and four IB ports per module in a POD configuration. In another example, the rack A9000 system has two InfiniBand switches connecting the modules to the back-end storage, while the POD A9000 system uses point-to-point connections between the modules and the back-end storage.

[0156] Additionally, in one aspect, the Integrated Management Controller (IMM) may include a dedicated board (CPU+memory+IO devices) within the A9000 module that continues to operate on standby power. The IMM may run its own operating system and a set of software services that allow for remote management and hardware diagnostics.

[0157] Further, in one aspect, assuming a UFI BIOS in communication with the IMM, the UFI system firmware running on the server CPU is responsible for booting the server. The firmware installs a set of exception handlers and timers that trigger the execution of UFI code when a hardware event occurs or when the timer is triggered. The UFI firmware preempts the execution of the operating system when its handlers are executing. The UFI firmware then communicates with the IMM and can request the IMM to perform error collection tasks and tasks to reset the server, shut down the server, or power cycle the server. When a hardware error occurs (such as a memory error or PCIE error), the UFI may collect data and attempt to determine whether to power down or power cycle the server. Depending on the failure scenario, these tasks may take a significant amount of time, resulting in modules with active IB links becoming inoperable.

[0158] Additionally, in an InfiniBand environment, Management Datagrams (MADs) are used to discover and configure the fabric and to execute remote hardware or software commands. Each MAD has a framing defined by class, method, attributes, and attribute qualifiers. The class defines the range of commands for use in a particular application, such as the SM (Subnet Manager) class and SA (Subnet Management) class for subnet managers, the performance class for monitoring, and the CM class for connection management. The method defines the type of access (for example, Get or Set). The attributes define the command within the class, and the qualifiers define the command parameters.

[0159] Also, in one aspect, a subnet manager comprises a software entity executing on one of the endpoints in the InfiniBand fabric. The subnet manager is responsible for discovering and configuring the InfiniBand fabric (e.g., by performing link management and configuration, unicast and multicast routing table switching, etc.) and providing subnet management services (e.g., multicast group management and fabric queries). The subnet manager implements discovery and configuration using SM MAD datagrams and SA services using SA MAD datagrams.

[0160] In another aspect, an active IB port includes an InfiniBand port that has a current IB link and is configured for data transfer using a high-speed, reliable protocol, also referred to as a port with an active IB link.

[0161] Table 1 includes an example crash detection API according to one example aspect.

[0162] [Table 1]

[0163] [Table 2]

[0164] The end_point_crash_detect_loop() API call monitors a single module (the server), and the context arguments define the monitoring parameters and addressing information.

[0165] The meaning of the context fields is as follows: params - the monitor parameters. The structure and monitor API are defined in such a way that monitor parameters can be changed from a thread context different from the monitor context. The meaning of the fields is as follows: probe_interval_ms - The probe interval (in milliseconds) for the first or first after a good ping failed_probe_interval_ms - The interval in milliseconds between probes after a failed ping is detected link_probe_to_ms - MAD timeout for link probe command os_probe_to_ms - MAD timeout for the OSping command link_reset_to_ms - MAD timeout for link reset command allowed_fail_cnt - the number of consecutive failed OS ping commands before a module link is reset (a failed ping is only counted as a hit if at least one of the links is detected as ACTIVE) allowed_bullets - number of resets allowed per bullet_lease_sec period bullet_lease_sec - a time span that defines how many resets (allowed_bullets) are allowed during this time. ·port_path - path parameter to the monitored module. Rack systems have two paths (they can probe both ports of their connection), POD systems have a single path (they can monitor a single port of their connection). The meaning of the fields in each path definition is as follows: dr_path_port - a direct path to a destination port. If you set and read paths using native aligned types (1, 2, 4, 8 bytes), this path can be safely modified from a thread context different from the monitoring context. The reason is that the monitoring code copies the path into a private data structure at the start of each monitoring cycle, and will use a consistent path definition both before and after the change. · dr_hop_cnt - defines the number of hops for a direct path. rem_port - specifies the port used to query the link. This port belongs to the previous hop of the monitored module. In RACK systems this is the same as the last hop in the address (switch port). In POD systems this is unknown and must be defined. · mad_handle - Specifies the local MAD device that should be used to reach the specified remote port. actual_fail_cnt - the number of consecutive failed OS ping probes - varies from 0 to allowed_fail_cnt. The value of this field is reset to 0 after the first successful OS ping and starts incrementing after the first unsuccessful OS ping (the value is incremented only if the OS ping fails and at least one of the links is active). used_bullets - The number of link resets (since bullet_fire_stamp) since the first allowed reset (including the first reset). Resets to 0 after bullet_lease_sec. bullet_fire_stamp - The time when the first link reset was issued within the allowed time window, or zero. os_probe_state_change_cb - callback that is called when the first ping succeeds or fails. Parameter probe_ok specifies whether the OS probe status should transition to failed or OK. A NULL value means that the callback will not be called. link_reset_cb - callback that is called when a decision is made to reset the link. The parameter reset_called specifies whether a reset was done or not (because the bullet is gone). A NULL value means that the callback will not be called. · os_probe_ok - status of the OS probe during the last iteration. · single_port - defines whether the monitored module has one path (point-to-point connection, as in the case of POD) or two paths (as in the case of RACK). The monitoring code takes this value into account. next_first_path - Defines which path index (0 or 1) to examine first during the next iteration. This field is updated by the monitoring code.

[0166] The context structure is defined to be easily accessible from different contexts, which allows a "trace" window into the monitoring context. The end_point_crash_detect_loop() API call continuously monitors a single connected module. The monitoring flow is as follows: Before monitoring begins, the monitoring parameters and addressing information are set in the monitoring context. The values ​​of the bullet used and the first bullet stamp are set to zero, as is the value of the next path index to probe. The value of os_probe_ok is set to true.

[0167] 1. When the monitor loop is started, it first copies the address information into a private data structure.

[0168] 2. The monitor loop then proceeds to test the OS status of the remote module, and the status or IB link of the remote module. The probe is performed using the direct path identified by the next path index (next_first_path) and the MAD handle. The OS probe is performed using the sm_info_ping() API with a timeout value equal to os_probe_to_ms. If the OS ping is successful, it is concluded that the remote port has link and the remote OS is alive. If the OS ping times out, the code proceeds to determine the state of the link. The link is verified using the query_port_info() API for the destination port rem_port, which is one hop back on the previously used path. The link verification uses the same MAD handle and a timeout value equal to link_probe_to_ms.

[0169] 3. The monitor code checks the remote module's probe status. If the OS probe is successful, the code proceeds to step 4. If the OS ping fails and the link probe returns failure, the code proceeds to step 5. The last decision is a design point that aims to minimize false positive detections and ensures that the closest side of the link (the switch in the case of a RACK system, or the module in the case of a POD system) is not the source of the MAD error. If the OS ping fails and the link probe returns success, the code proceeds to step 6.

[0170] 4. If the OS ping status is successful, the code checks whether to call the os_probe_state_change_cb callback. The callback is called if its value is not NULL and the value of os_probe_ok is false. The code then proceeds to step 5.

[0171] 5. In this step, the code prepares to start a new monitoring cycle. Unless single_port is specified, the code flips the value of next_first_path (0 → 1 and 1 → 0). Then the code sets os_probe_ok to a true value, sets the actual_fail_cnt field to zero, restarts the monitoring loop after probe_interval_ms milliseconds, and returns to step 1.

[0172] 6. If the link probe was successful, the code checks the state of the link. If the link is active, the code proceeds to step 7. If the link is not active, the code proceeds to step 8.

[0173] 7. In this step, the code has determined that the link state is active, and there may be a crashed OS with an active link. The code then checks whether there are any additional paths to the monitored module to check. If the single_port parameter is set or other paths have already been checked, no additional paths are available. If there are no additional paths to check, the code proceeds to step 9. If additional paths are available, the code proceeds to step 8.

[0174] 8. This step is only performed on RACK systems that monitor modules using two paths. In this step, the code has determined that the OS probe failed over the first path and the link on the first path is either down or active. This is not sufficient to determine a recovery action, since the monitoring logic needs to reliably detect the problem over both ports of the monitored module. Therefore, the code resets the value of next_first_path and proceeds to step 2.

[0175] 9. In this step, the code sets the value of os_probe_ok to false and checks if the os_probe_state_change_cb callback should be called. The callback is called if its value is not NULL and the value of os_probe_ok is true. The code then proceeds to the next step, step 10.

[0176] 10. In this step, the code determines whether the monitored module is a crashed module with an active IB link. If at least one of the links connecting to the monitored module is in an active state, the module is a crashed module with an active IB link. If the module is not a crashed module with an active IB link, the code resets the actual_fail_cnt field to zero and proceeds to step 16. Otherwise, the code proceeds to step 11.

[0177] 11. In this step, the code determines that a crashed module with an active IB link has been detected. The code increments the count in actual_fail_cnt. If the incremented value is less than the value in allowed_fail_cnt, the code proceeds to step 16. Otherwise, the code proceeds to step 12.

[0178] 12. At this point, the code has determined the recovery action (a link reset is required). The code resets the actual_fail_cnt field to zero and checks whether the used_bullets value is less than the allowed_bullets value. If the used_bullets value is less than the allowed_bullets value, the code proceeds to step 15. Otherwise, the code proceeds to step 13.

[0179] 13. In this step, the code checks whether the current timestamp is greater than bullet_fire_stamp+bullet_lease_sec. If the current timestamp value is greater, the code resets the used_bullets value to zero and proceeds to step 15. Otherwise, the code proceeds to step 14.

[0180] 14. In this step, if the value of the callback is not NULL, the code executes the link_reset_cb callback with the value of the reset_called parameter set to false and proceeds to step 16.

[0181] 15.In this step the code performs recovery actions and resets the IB links on the crashed module. All IB links on the monitored modules are reset using the provided path information. The same path and MAD handles are used as in the link state query. The code uses the reset_port() API with a timeout value of link_reset_to_ms. The API uses the Set method of the SM PortInfo MAD to disable and then enable the physical properties of the links, causing a link reset. The physical link reset propagates quickly throughout the logical (software) layer, allowing the application to reset connections and flush pending IO transactions. Additionally, the A9000 SM component does not enable the logical properties of links on non-responsive modules, so the logical links on the crashed module remain down. The reset method of recovery, as opposed to link disabling, is a design point to minimize the impact of false positive detections. If the code resets links on a busy but alive module, the SM component will reactivate the logical properties of the IB links once the physical links have been negotiated and the software on the monitored module starts responding. Following this step the code proceeds to step 16.

[0182] 16. In this step, the code resumes monitoring with a reduced monitor timeout. Unless single_port is specified, the code flips the value of next_first_path (0 -> 1 and 1 -> 0). The code then resumes the monitoring loop after failed_probe_interval_ms milliseconds and goes back to step 1.

[0183] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of integration of technical details. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0184] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0185] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0186] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform embodiments of the present invention.

[0187] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to aspects of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0188] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0189] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that causes a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0190] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than that described in the figures. For example, two blocks shown in succession may in fact be accomplished as one step, or may be performed simultaneously, substantially simultaneously, in a partially or fully overlapping manner in time, or in some cases the blocks may be performed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0191] Further, systems according to various aspects may include a processor and logic integrated with and / or executable by the processor, the logic configured to perform one or more of the process steps described herein. Integrated means that the logic is embedded in the processor as hardware logic, such as an application specific integrated circuit (ASIC), FPGA, etc. Executable by the processor means that the logic is hardware logic, software logic, such as firmware, part of an operating system, part of an application program, or a combination of hardware logic and software logic, accessible by the processor and configured to cause the processor to perform some function when executed by the processor. The software logic may be stored in any memory type, local and / or remote memory, as known in the art. Any processor known in the art may be used, such as a software processor module and / or hardware processor, such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), graphics processing unit (GPU), etc.

[0192] It will be apparent that various features of the above-described systems and / or methodologies can be combined in any manner to create multiple combinations from the description provided above.

[0193] It will be further appreciated that embodiments of the present invention may be provided in the form of a service that is deployed on behalf of a customer to provide the service on demand.

[0194] The description of various aspects of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed aspects. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described aspects. The terms used in this specification are selected to best explain the principles of the aspects, practical applications, or technical improvements beyond the technology found in the market, or to enable those skilled in the art to understand the aspects disclosed herein.

Claims

1. 1. A computer-implemented method comprising: determining whether an operating system of a second node of the distributed computing environment is functioning properly by sending a first management query from the first node to the second node; in response to determining that the operating system of the second node is not functioning properly, determining whether the second node has an active communications link by sending a second management query from the first node to a port associated with the second node; in response to determining that the second node has an active communication link, resetting the active communication link of the second node by transmitting a reset request from the first node to the port associated with the second node; and A computer-implemented method, comprising: a plurality of additional nodes included in the distributed computing environment along with the first node and the second node; and each of the plurality of additional nodes in the distributed computing environment monitors adjacent nodes utilizing the first management query and the second management query.

2. 2. The computer-implemented method of claim 1, wherein the first management query is sent directly from the first node to the second node of the distributed computing environment via a point-to-point connection between the first node and the second node.

3. 2. The computer-implemented method of claim 1, wherein the first management query is sent indirectly from the first node to the second node of the distributed computing environment via a switch disposed between the first node and the second node.

4. The computer-implemented method of claim 1, wherein in response to determining that the second node is an adjacent node of the first node, the first management query is sent by the first node to the second node.

5. 2. The computer-implemented method of claim 1, wherein the first management query comprises a fabric management query that terminates within the operating system of the second node.

6. 2. The computer-implemented method of claim 1, wherein in response to determining that a response to the first management query was not received from the second node within a predetermined period of time, it is determined that the operating system of the second node is not functioning properly.

7. 2. The computer-implemented method of claim 1, wherein the second management query queries the port associated with the second node whether the second node has an active communication link.

8. 2. The computer-implemented method of claim 1, wherein the second node is determined to have an active communication link in response to identifying an active port in the second node in response to the second management query.

9. 2. The computer-implemented method of claim 1, wherein in response to determining that the operating system of the second node is not functioning properly and the second node has an active communications link, a predetermined count is incremented and monitoring of the second node is performed more frequently.

10. 2. The computer-implemented method of claim 1, wherein the reset request causes the port associated with the second node to reset all active links within the second node.

11. The computer-implemented method of claim 1 , wherein each node in the distributed computing environment is designated as a monitor node.

12. 2. The computer-implemented method of claim 1, wherein a monitor node in the distributed computing environment is only permitted to send a predetermined number of reset requests to another node within a predetermined period of time.

13. The computer-implemented method of claim 1 , wherein the first management query and the second management query are both high priority queries.

14. 2. The computer-implemented method of claim 1, wherein the first management query also determines whether a network adapter of the second node is operational.

15. 15. A computer program product for isolating non-responsive ports in a network fabric, the computer program product causing a processor of the first node to perform the computer-implemented method of claim 1.

16. 1. A computer-implemented method comprising: transmitting a first management query from a first node of a distributed computing environment to a second node of the distributed computing environment, the first management query being associated with an operating system of the second node; in response to failure to receive a response to the first management query from the second node at the first node, transmitting a second management query from the first node to one or more communication ports associated with the second node; in response to receiving, at the first node, from the one or more communication ports associated with the second node a response to the second management query indicating that the second node has an active communication link, transmitting, by the first node, a link reset request to the one or more communication ports associated with the second node requesting that the one or more communication ports of the second node having the active communication link be reset; and A computer-implemented method, comprising: a plurality of additional nodes included in the distributed computing environment along with the first node and the second node; and each of the plurality of additional nodes in the distributed computing environment monitors adjacent nodes utilizing the first management query and the second management query.

Images