A distributed storage environment network sub-health detection and fault automatic processing method

Abstract

The present invention relates to a distributed storage environment network sub-health detection and automatic fault processing method, which is used for network sub-health detection and fault processing of a distributed storage system, wherein the distributed storage system includes a storage cluster and a central management cluster, wherein the storage cluster includes multiple storage nodes, and comprises the following steps: obtaining the network configuration status of the storage cluster through cluster view perception; cyclically obtaining the node status detection result reported by each storage node, wherein the node status detection result is obtained by each storage node based on an active detection mechanism, including network status information of other storage nodes except its own storage node; based on the node status detection result within a set time, determining whether there is a storage node with network sub-health, and if so, executing network repair or isolation processing of the corresponding node based on the network configuration status. Compared with the prior art, the present invention has the advantages of reducing operation and maintenance costs and reducing operation and maintenance complexity.

Description

Technical Field

[0001] The present invention relates to a network health detection technology in a distributed storage environment, and in particular to a network sub-health detection and automatic fault processing method in a distributed storage environment. Background Art

[0002] In a distributed storage environment, the physical or software modules involved in the entire data read and write logic include: servers, switches, common network cables, common network card modules, optical fiber cables, optical fiber modules, operating systems, etc. Among them, the network-related modules serve as a bridge for data exchange in the entire storage system. When a network module fails, it may affect the storage system response at the least, or cause a cascade reaction at the worst, causing the entire storage system to be unable to provide external services.

[0003] Among the above network abnormalities, network sub-health is an objective but ambiguous situation, that is, although it can continue to work, it may seriously affect the business. Common network sub-health includes packet loss, delay, congestion, etc. When the above sub-health state occurs, since the network is not completely inoperable, but is barely running in an inefficient mode, some storage nodes become bottlenecks, affecting the response time of the entire storage system. Therefore, in a high-quality distributed storage system, the detection and troubleshooting of network sub-health is an indispensable part of improving the stability of storage products.

[0004] Currently, there are two common network sub-health detection mechanisms: 1. Check the statistical values ​​fed back by the network card itself or the switch, such as monitoring indicators such as packet loss rate and packet retransmission rate, and determine whether the network card or link is abnormal through the statistical results; 2. Indirectly determine whether the network card or link is abnormal by building statistical values ​​of actual business on the distributed storage system, such as monitoring indicators such as request response time and number of error requests. The above methods have the following disadvantages:

[0005] 1. Relying on the monitoring indicators of the network card hardware or switch hardware, different manufacturers have different definitions, which leads to the need for distributed storage software to adapt to the hardware and firmware of multiple manufacturers, resulting in high development costs and poor universality.

[0006] 2. Relying on the monitoring indicators of the actual business will cause the actual business to consume more resources for monitoring network sub-health, etc., and not all upper-level businesses have their own monitoring functions. For distributed storage systems, it is impossible to achieve universal adaptation effects.

[0007] 3. Whether relying on network card hardware or switches or relying on the monitoring of actual business, there is a lag, that is, only when network sub-health has occurred and the network card hardware, switches, and actual business have sensed the abnormality will it be defined as network sub-health. At this time, it has actually affected the upper-level applications and cannot be perceived in advance.

[0008] The common network sub-health fault handling mechanism is: when monitoring finds a network sub-health event, it is necessary to manually switch the server redundant network card, and then observe the subsequent monitoring indicators to determine whether it has recovered, so as to respond. This operation has the following disadvantages:

[0009] 1. When network sub-health occurs, manual response takes time.

[0010] 2. In a distributed storage cluster, the number of storage nodes ranges from dozens to hundreds, which will result in long manual operation time and many operation steps, and even potential misoperation, causing secondary failures. Summary of the invention

[0011] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for sub-health detection and automatic fault handling in a distributed storage environment. In a distributed storage environment, the network status of the entire cluster is perceived through the cluster view, so as to more accurately determine the faulty node and handle the fault more quickly, shorten the storage cluster failure time, reduce operation and maintenance costs, and reduce operation and maintenance complexity.

[0012] The purpose of the present invention can be achieved by the following technical solutions:

[0013] A distributed storage environment network sub-health detection and automatic fault processing method is used to perform network sub-health detection and fault processing on a distributed storage system, wherein the distributed storage system includes a storage cluster and a central management cluster, and the storage cluster includes multiple storage nodes. The method includes the following steps:

[0014] Obtain the network configuration status of the storage cluster through cluster view perception;

[0015] Circularly obtain the node status detection results reported by each storage node. The node status detection results are obtained by each storage node based on the active detection mechanism, including the network status information of other storage nodes except its own storage node;

[0016] Based on the node status detection result within the set time, it is determined whether there is a storage node with network sub-health. If so, network repair or isolation processing of the corresponding node is performed based on the network configuration status.

[0017] Further, the network configuration state is received from each storage node, and each storage node generates the network configuration state through the following steps:

[0018] Get the service IP address of the current storage node configuration, get the network card name corresponding to the IP address, determine whether the network card is a logical network card, if so, get the names of multiple physical network cards actually corresponding to the logical network card, get the configuration attributes and status of the logical network card and the corresponding physical network card, record the name of the primary network card actually used by the current logical network card, and generate the network configuration status; if not, directly record the physical network card name and generate the network configuration status.

[0019] Furthermore, the storage node obtains the node status detection result based on the active detection mechanism as follows:

[0020] Each storage node periodically obtains a list of all storage nodes in the current storage cluster;

[0021] Each storage node periodically initiates a network detection request to other storage nodes using its currently used IO network as the source address, and generates the node status detection result according to the request response of other storage nodes.

[0022] Furthermore, the initiating network detection request is specifically:

[0023] Within the specified time, try to send as many ICMP data packets as possible to other storage nodes, count the number of response data packets received within the specified time and the sending and receiving time of each response data packet, and finally obtain the packet loss rate and delay from the current storage node to all other storage nodes per unit time as the node status detection result.

[0024] Furthermore, at the same time, each storage node sends the ICMP data packet to other storage nodes in parallel.

[0025] Furthermore, if within a set time, the packet loss rate and delay from a storage node to other storage nodes exceed the set threshold and last for a specified period of time, it is determined that the network sub-health of the storage node occurs.

[0026] Furthermore, the network repair specifically includes: switching the physical network card corresponding to the logical network card.

[0027] Furthermore, the isolation process specifically includes: shutting down the IO logic storage service of the storage node or shutting down the power supply.

[0028] Furthermore, for a storage node that shuts down its processing IO logical storage service, the network status information of the storage node is continuously obtained to determine whether the network sub-health status of the storage node disappears. If so, the processing IO logical storage service of the storage node is restored.

[0029] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the distributed storage environment network sub-health detection and automatic fault handling method as described.

[0030] Compared with the prior art, the present invention performs global statistics and analysis on the network status of multiple nodes in real time in a distributed scenario, so that network sub-health can be discovered in advance in the scenario of distributed storage regardless of whether there is business operation or whether the business load is high, and abnormal node network failure can be automatically handled. If the network sub-health cannot be eliminated after handling the abnormality, the abnormal node is isolated to ensure the availability and performance indicators of the entire distributed storage cluster, which has the following beneficial effects:

[0031] (1) The present invention can run independently of the distributed system software itself and can be adapted to a general distributed storage system without the need for secondary development or docking of the storage system.

[0032] (2) The present invention does not require any monitoring logic of the upper-layer business, and the upper-layer business does not require secondary development or docking.

[0033] (3) The present invention uses a cluster view to check and analyze the entire network status, and can locate faulty nodes more accurately.

[0034] (4) The present invention is an automatic detection and early perception, which does not require the actual operation of the business or the business is not running at full load, and can discover network sub-health problems in advance.

[0035] (5) After detecting that the network is sub-healthy, the present invention can attempt to automatically repair the distributed storage cluster in multiple dimensions by switching logical network cards or isolating physical nodes.

[0036] (6) The present invention does not require human intervention throughout the entire process, which reduces response time, operation and maintenance costs, and operation and maintenance complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure of the distributed storage system of the present invention;

[0038] Figure 2 It is a schematic diagram of the decomposition of the execution process of the present invention;

[0039] Figure 3 It is a schematic diagram of the process of network sub-health detection and fault handling of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] The present embodiment provides a method for detecting sub-health in a distributed storage environment and automatically handling faults, which is used for detecting sub-health in a distributed storage system and handling faults in a distributed storage system. The distributed storage system includes a storage cluster and a central management cluster. The storage cluster includes multiple storage nodes. The method includes the following steps: obtaining the network configuration status of the storage cluster through cluster view perception; cyclically obtaining the node status detection results reported by each storage node, which are obtained by each storage node based on an active detection mechanism, including network status information of other storage nodes except its own storage node; based on the node status detection results within a set time, determining whether there is a storage node with sub-health in the network, and if so, performing network repair or isolation processing of the corresponding node based on the network configuration status. Since the node status detection results reported by each storage node are cyclically obtained, it is possible to determine whether the network sub-health has been restored based on the detection results reported by each subsequent storage node.

[0042] A distributed storage system in which the above method is applied is as follows Figure 1 As shown, it includes a storage cluster, a central management cluster and a business cluster. The storage cluster contains multiple storage nodes, each storage node is connected to the central management cluster through multiple switches, and the business cluster includes multiple block storage access clients. Each storage node has a service for processing IO logical storage services, configuring storage node services and monitoring storage node services. Each storage node corresponds to a logical network card or a physical network card, and the logical network card is composed of multiple physical network cards.

[0043] like Figure 3 As shown, the above method counts the packet loss and delay in the recent specified time period to determine whether it exceeds the set threshold. If so, it is determined that there is a storage node in a sub-healthy state of the network, and the network card switching operation is performed first. If the network sub-health is eliminated, the process ends. If not, it is determined whether the node needs to be shut down. If so, the server power is turned off. If not, the IO logic storage service is shut down.

[0044] In the above method, the network configuration status of the storage cluster is firstly sensed through the cluster view, including sensing the network configuration of the storage nodes in the storage cluster, identifying the network card binding type and the status of each sub-card, specifically:

[0045] 201) Identify the IP address configured for the storage node;

[0046] 202) Find out the corresponding network card name through the IP address;

[0047] 203) Determine whether the network card is a logical network card, that is, whether it is obtained by binding multiple physical network cards;

[0048] 204) Obtain a list of physical network cards corresponding to the logical network card;

[0049] 205) Obtain the physical network card that actually corresponds to the logical network card in the current state.

[0050] The above steps can be implemented by the monitoring storage node service of the storage node.

[0051] In the above method, the process of each storage node in the storage cluster implementing the active detection mechanism is as follows: Figure 2 As shown, taking a storage node as an example, the specific steps include:

[0052] As shown in process 1, storage node 1 obtains a list of other storage nodes from the central management cluster. Assume that there are n storage nodes in the storage cluster.

[0053] As shown in processes 2.1, 2.2 and 2.3, storage node 1 periodically initiates network detection requests to other n-1 storage nodes using its currently used IO network as the source address, and generates node status detection results based on the request responses of other storage nodes. 2.1, 2.2 and 2.3 are initiated simultaneously to ensure that other nodes detect at the same time.

[0054] As shown in process 3, storage node 1 reports the node status detection result obtained in the previous step to the central management cluster. At this time, storage node 1 completes one cycle of monitoring result reporting logic and repeats the above processes 1, 2, and 3 in the next cycle.

[0055] As shown in process 4, the central management cluster collects and analyzes the network health status reported by each storage node, counts and analyzes the storage nodes with sub-healthy networks. If the packet loss rate and delay from a storage node to other storage nodes exceed the set threshold within the set time and continue for a specified period of time, the storage node is judged to have sub-healthy networks.

[0056] Assuming that the network of storage node 1 is sub-healthy, the central management cluster automatically initiates network repair or isolates the sub-healthy network node. If network repair is selected, the central management cluster sends a network card switching request to storage node 1. After the storage node is switched, the logical network card is changed from the initial state of process 0 (the logical network card points to network card 1) to point to network card 2 according to process 5. If the network sub-health anomaly is eliminated after the switch, the fault is eliminated and the process ends. If the network sub-health anomaly is not eliminated after the switch, the storage node removal operation is initiated, as shown in process 6. The storage node removal operation can isolate the node as needed, specifically including: shutting down the storage node's processing IO logical storage service or turning off the power supply.

[0057] For storage nodes that shut down the processing IO logical storage service of the storage node, the network status information of the storage node is continuously obtained to determine whether the network sub-health status of the storage node disappears, that is, the packet loss rate and delay are restored to below the threshold. If so, the processing IO logical storage service of the storage node is restored.

[0058] The central management cluster determines whether the network sub-health has recovered based on the detection results reported by each storage node.

[0059] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0060] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for sub-health detection and automatic fault handling of a distributed storage environment network, characterized in that: The method is used for performing network sub-health detection and fault handling on a distributed storage system, wherein the distributed storage system includes a storage cluster and a central management cluster, and the storage cluster includes multiple storage nodes. The method includes the following steps: Obtain the network configuration status of the storage cluster through cluster view perception; Circularly obtain the node status detection results reported by each storage node. The node status detection results are obtained by each storage node based on the active detection mechanism, including the network status information of other storage nodes except its own storage node; Based on the node status detection results within the set time, determine whether there is a storage node with sub-healthy network, and if so, perform network repair or isolation processing of the corresponding node based on the network configuration status; The network configuration state is received from each storage node, and each storage node generates the network configuration state through the following steps: Obtain the service IP address of the current storage node configuration, obtain the network card name corresponding to the IP address, determine whether the network card is a logical network card, if so, obtain the names of multiple physical network cards actually corresponding to the logical network card, and obtain the configuration attributes and status of the logical network card and the corresponding physical network card, record the name of the primary network card actually used by the current logical network card, and generate the network configuration status; if not, directly record the physical network card name and generate the network configuration status; The storage node obtains the node status detection result based on the active detection mechanism as follows: Each storage node periodically obtains a list of all storage nodes in the current storage cluster; Each storage node periodically initiates a network detection request to other storage nodes using its currently used IO network as the source address, and generates the node status detection result according to the request response of other storage nodes; The initiation of the network detection request is specifically: Try to send as many ICMP packets as possible to other storage nodes within the specified time, count the number of response packets received within the specified time and the sending and receiving time of each response packet, and finally obtain the packet loss rate and delay from the current storage node to all other storage nodes per unit time as the node status detection result; If within the set time, the packet loss rate and delay from a storage node to other storage nodes exceed the set threshold and continue for a specified period of time, the storage node is judged to have network sub-health.

2. The distributed storage environment network sub-health detection and automatic fault handling method according to claim 1 is characterized in that: At the same time, each storage node sends the ICMP data packet to other storage nodes in parallel.

3. The distributed storage environment network sub-health detection and automatic fault handling method according to claim 1 is characterized in that: The network repair specifically includes: switching the physical network card corresponding to the logical network card.

4. The distributed storage environment network sub-health detection and automatic fault handling method according to claim 1 is characterized in that: The isolation process specifically includes: shutting down the IO logic storage service of the storage node or shutting down the power supply.

5. The method for sub-health detection and automatic fault handling in a distributed storage environment network according to claim 4, characterized in that: For a storage node that shuts down its processing IO logical storage service, the network status information of the storage node is continuously obtained to determine whether the network sub-health status of the storage node disappears. If so, the processing IO logical storage service of the storage node is restored.

6. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the distributed storage environment network sub-health detection and automatic fault handling method as described in any one of claims 1-5.

Images