Storage protocol and network layer fault processing module and method and fault sensing method and device

Abstract

The invention discloses a storage protocol and network layer fault processing module and method, and a fault sensing method and device, and belongs to the field of storage network fault sensing and processing, and the method comprises the steps: a layer of fault registration management module is newly added in a Linux kernel, and the module is used for recording iSCSI connection and iSER connection, and recording related callback functions of an upper layer at the same time; and when the Linux Bond has a problem, calling a callback function recorded by the fault registration management module according to a triggered error event to complete fault processing. The invention provides a high availability solution at a storage network layer.

Description

Technical Field

[0001] The present invention relates to the field of storage network fault perception and processing technology, and more specifically, to a storage protocol and network layer fault processing module and method, and a fault perception method and device. Background Art

[0002] In traditional storage networks, Linux Bonding technology is often used to provide link redundancy solutions. It can provide functions such as load balancing and network redundancy. However, different problems still exist when storage protocols use Linux Bonding networks.

[0003] Although Linux Bonding can provide network redundancy, the upper-layer storage protocol cannot quickly sense the network interruption when the network is completely disconnected. The commonly used method is to use a timed ping-pong mechanism on the client or storage end to detect the health status of the network. This mechanism generally takes too long and often affects storage services when the network is completely disconnected.

[0004] In the RDMA network scenario, each RDMA network card has a send queue and a receive queue. The send queues and receive queues of different network cards cannot be shared. In addition, the current Linux Bonding mechanism cannot obtain the send queue and receive queue resources of two different network cards. As a result, when one network card is interrupted, the network request cannot be quickly sent to another network card, which leads to the interruption of the entire network. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art, provide a storage protocol and network layer fault processing module and method, a fault perception method and device, and provide a high availability solution at the storage network layer, which can solve the problem that Linux Bonding cannot provide high availability on the RDMA network card, and can solve the problem that the network interruption detection time of Linux Bonding is too long.

[0006] The object of the present invention is achieved through the following solutions:

[0007] A storage protocol and network layer fault processing module, comprising:

[0008] A new fault registration management module is added to the Linux kernel. This module is used to record iSCSI connections and iSER connections, and also records the related callback functions of the upper layer.

[0009] When a problem occurs in Linux Bond, the callback function recorded in the fault registration management module is called according to the triggered error event to complete the fault processing.

[0010] A storage protocol and network layer fault processing method comprises the following steps:

[0011] S1, add an error handling callback function linked list in the Linux kernel, the elements in the linked list are used to record: iSCSI connection resources of each storage protocol layer, completion callback functions related to the connection resources, related fault activity events, and unique identifiers of different resource devices under the bound device and device-related callback functions;

[0012] S2, when the Linux Bond device receives a link failure of a single network card or the entire bonded device, different activity events will occur, and the callback function of the network device will be triggered according to the type of the corresponding activity event.

[0013] Furthermore, in step S2, triggering the callback function of the network device according to the type of the corresponding activity event specifically includes the following sub-steps:

[0014] S21, in the callback function of the device, determine the corresponding iSCSI connection resource according to the resource ID of the network resource device, and then access the completion callback function corresponding to the connection resource;

[0015] S22, in the completion callback function of the connection resource, calling the connection closing function to close the iSCSI connection;

[0016] S23, after the iSCSI initiator finds that the connection of the target end is disconnected, it uses the original mechanism of its kernel to trigger the re-login connection reconnection mechanism and re-establish a new connection.

[0017] A storage protocol and network layer fault perception method, which establishes a connection relationship between a storage protocol layer and a network device layer based on the storage protocol and network layer fault processing module as described above;

[0018] The network device failure is then indexed to the corresponding storage protocol layer resources, and the connection at the storage protocol layer is closed to make the client aware of it.

[0019] Furthermore, the method of indexing the failure of the network device to the resources of the corresponding storage protocol layer and closing the connection of the storage protocol layer so as to make the client aware of it specifically includes the following sub-steps:

[0020] S31, if it is an RDMA network card device, when a single path is disconnected, the Linux Bond device is considered unusable, which triggers the failure of the entire network card device;

[0021] S32, determining all iSCSI connections corresponding to the device, and calling their corresponding iSCSI connection completion callback functions respectively;

[0022] S33, after passing through the storage protocol and network layer fault processing modules, the iSCSI initiator client will receive an error code of an unknown error from the server, which will trigger a reconnection request;

[0023] S34, after reconnecting to the target end, re-establishing the socket connection or RDMA connection channel resources to the bound device, the bound device re-establishes the request to the available network card.

[0024] Furthermore, the method of indexing the failure of the network device to the resources of the corresponding storage protocol layer and closing the connection of the storage protocol layer so as to make the client aware of it specifically includes the following sub-steps:

[0025] S41, if it is a TCP network card device, when a single path fails, use the Linux Bond function to switch to an available network card device;

[0026] S42, when the entire Bond device fails, all iSCSI connections corresponding to the device are judged, and the corresponding iSCSI connection completion callback functions are called respectively; after passing through the storage protocol and network layer fault handling modules, the iSCSI initiator client will receive an error code of an unknown error on the server side, which will trigger a reconnection request; finally, after reconnecting to the target again, the socket connection or RDMA connection channel resources are re-established to the bound device, and the bound device re-establishes the request to the available network card.

[0027] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described above is executed.

[0028] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in any one of the above items is executed.

[0029] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in any one of the above items is executed.

[0030] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described above is executed.

[0031] The beneficial effects of the present invention include:

[0032] The embodiment of the present invention provides a high availability solution at the storage network layer, which can solve the problem that Linux Bonding cannot provide high availability when a single path of an RDMA network card fails, and can solve the problem that when all LinuxBondings in a TCP network card are interrupted, the transmission network interruption detection time is too long, which leads to a long high availability switching time of the storage protocol layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 is a working diagram of a fault registration management module in an embodiment of the present invention;

[0035] Figure 2 A storage protocol link list in an embodiment of the present invention;

[0036] Figure 3 A flowchart of the steps of the storage protocol and network layer fault handling method;

[0037] Figure 4 Flowchart of steps to register a module for a fault;

[0038] Figure 5 A flowchart of the steps for storage protocol and network layer fault perception in the device scenario of RDMA network card;

[0039] Figure 6 A flowchart of the steps for storage protocol and network layer fault perception in the device scenario of a TCP network card. DETAILED DESCRIPTION

[0040] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0041] Explanation of terms

[0042] Linux Bonding, a network bonding technology, provides link redundancy, bandwidth multiplication and load balancing functions for the network;

[0043] RQ, Receive Queue, receive queue;

[0044] SQ, Send Queue, sending queue;

[0045] QP, Queue Pair, queue pair, RDMA work requests will be passed through the receive queue and the send queue, and when they are completed, they will be placed in the CQ for completion;

[0046] iSCSI, Internet Small Computer System Interface, small computer system interface;

[0047] iSER, iSCSI Extensions for RDMA, extension of iSCSI interface in RDMA;

[0048] iSCSI Target, the target of iSCSI service;

[0049] iSER Target, the target of iSER service;

[0050] iSCSI Session, iSCSI connection;

[0051] iSER Session, iSCSI connection;

[0052] error_handler_resource, error handling callback function;

[0053] event, event;

[0054] session, connection;

[0055] complete_callback, completion callback function;

[0056] callback, callback function;

[0057] bond, bind;

[0058] iSCSI initiator, iSCSI initiator;

[0059] target, target device;

[0060] login reconnect, re-login connection;

[0061] close function, close function;

[0062] iSCSI session complete_callback, iSCSI connection completion callback function;

[0063] server, server side;

[0064] reconnect, reconnect;

[0065] socket fd, socket connection;

[0066] rdma channel, RDMA connection channel.

[0067] The specific implementation process of the present invention is as follows:

[0068] In one embodiment of the present invention, a storage protocol and network layer fault handling module are provided. The main problem in the native Linux kernel is that the Linux Bond device can quickly sense the failure of one or more network devices. The main reason is that after sensing the failure, the data stream related to the failed network device is switched to a good network device. However, when all devices fail, that is, when the Linux Bond device cannot work, its error cannot be sensed by the upper iSCSI Target or iSER Target, which causes the upper application to be unable to quickly switch the path device.

[0069] A new fault registration management module is added to the original Linux kernel module. This kernel module is mainly used to record all iSCSI Session or iSER Session connections, and also record the upper layer related callback functions. When a Linux Bond has a problem, a related error event will be triggered, and the callback function will be called according to the error event.

[0070] In another embodiment of the present invention, a specific workflow of a storage protocol and network layer fault processing module is provided, including the following steps:

[0071] Add an error_handler_resource linked list in the kernel. The elements in the linked list are used to record the iSCSI session resources of each storage protocol layer, the complete_callback callback function related to the session resource, the related fault event, the unique identifier of different resource devices under the Bond device and the callback function related to the device.

[0072] 1) When the Linux Bond device receives a link failure of a single network card or the entire bond device, different events will occur. The callback function of the network device will be triggered according to the type of event.

[0073] 2) In the callback function of the device, the corresponding iSCSI session resource is determined according to the resource ID of the network resource device, and then the complete_callback function corresponding to the session resource is accessed.

[0074] 3) In the complete_callback function of the session resource, call the close function of Session to close the iSCSI session connection.

[0075] 4) After the iSCSI initiator finds that the session on the target is disconnected, it uses the original mechanism of its kernel to trigger the login reconnect mechanism and re-establish a new connection.

[0076] In another embodiment of the present invention, a method for rapid fault perception of a storage protocol and a network layer is provided. Since a connection relationship between the storage protocol layer and the network device layer is established, the failure of the network device can be quickly indexed to the corresponding storage protocol layer resources, and the client can be quickly perceived by quickly closing the connection of the storage protocol layer.

[0077] 1) Processing strategy for RDMA network card devices

[0078] a) Since the network resources such as the sending queue and receiving queue of the RDMA network card cannot be shared, when a single path is disconnected, the Linux Bond device is considered unusable, which triggers the failure of the entire network card device.

[0079] b) Determine all iSCSI session connections corresponding to the device and call their corresponding iSCSI sessioncomplete_callback functions respectively.

[0080] c) After passing through the storage protocol and network layer fault handling modules, the iSCSI initiator client will receive an error code indicating an unknown error from the server, which will trigger a reconnect request.

[0081] d) After reconnecting to the target, the socket fd or rdmachannel resources will be re-established to the bond device, and the bond device will re-establish the request to the available network card, thereby achieving the effect of fast failover.

[0082] 2) Processing strategy for TCP network card devices

[0083] a) Since TCP network card devices do not require the hardware resource information of the network card and all resources are shared in the kernel state, when a single path fails, the Linux Bond function can be used to quickly switch to an available network card device.

[0084] b) When the entire Bond device fails, the solutions 1-b, 1-c, and 1-d can be used. That is,

[0085] Determine all iSCSI session connections corresponding to the device and call their corresponding iSCSI sessioncomplete_callback functions respectively.

[0086] After passing through the storage protocol and network layer fault handling modules, the iSCSI initiator client will receive an error code indicating an unknown error from the server, which will trigger a reconnect request.

[0087] Finally, after reconnecting to the target, the socket fd or rdmachannel resources will be re-established to the bond device, and the bond device will re-establish the request to the available network card, thereby achieving the effect of fast failover.

[0088] It should be noted that within the scope of protection defined in the claims of the present invention, the following embodiments can be combined and / or expanded or replaced in any logical way from the above specific implementation methods, such as disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0089] Example 1

[0090] A storage protocol and network layer fault processing module, comprising:

[0091] A new fault registration management module is added to the Linux kernel. This module is used to record iSCSI connections and iSER connections, and also records the related callback functions of the upper layer.

[0092] When a problem occurs in Linux Bond, the callback function recorded in the fault registration management module is called according to the triggered error event to complete the fault processing.

[0093] Example 2

[0094] A storage protocol and network layer fault handling method, such as Figure 3 As shown, the following steps are included:

[0095] S1, add an error handling callback function linked list in the Linux kernel, the elements in the linked list are used to record: iSCSI connection resources of each storage protocol layer, completion callback functions related to the connection resources, related fault activity events, and unique identifiers of different resource devices under the bound device and device-related callback functions;

[0096] S2, when the Linux Bond device receives a link failure of a single network card or the entire bonded device, different activity events will occur, and the callback function of the network device will be triggered according to the type of the corresponding activity event.

[0097] Example 3

[0098] On the basis of Example 2, Figure 4 As shown, in step S2, triggering the callback function of the network device according to the type of the corresponding activity event specifically includes the following sub-steps:

[0099] S21, in the callback function of the device, determine the corresponding iSCSI connection resource according to the resource ID of the network resource device, and then access the completion callback function corresponding to the connection resource;

[0100] S22, in the completion callback function of the connection resource, calling the connection closing function to close the iSCSI connection;

[0101] S23, after the iSCSI initiator finds that the connection of the target end is disconnected, it uses the original mechanism of its kernel to trigger the re-login connection reconnection mechanism to re-establish a new connection.

[0102] Example 4

[0103] A storage protocol and network layer fault perception method, which establishes a connection relationship between a storage protocol layer and a network device layer based on the storage protocol and network layer fault processing module described in Example 1;

[0104] The network device failure is then indexed to the corresponding storage protocol layer resources, and the connection at the storage protocol layer is closed to make the client aware of it.

[0105] Example 5

[0106] On the basis of Example 4, Figure 5 As shown, the failure of the network device is indexed to the resources of the corresponding storage protocol layer, and the connection of the storage protocol layer is closed to make the client aware of it, which specifically includes the following sub-steps:

[0107] S31, if it is an RDMA network card device, when a single path is disconnected, the Linux Bond device is considered unusable, which triggers the failure of the entire network card device;

[0108] S32, determining all iSCSI connections corresponding to the device, and calling their corresponding iSCSI connection completion callback functions respectively;

[0109] S33, after passing through the storage protocol and network layer fault processing modules, the iSCSI initiator client will receive an error code of an unknown error from the server, which will trigger a reconnection request;

[0110] S34, after reconnecting to the target end, re-establishing the socket connection or RDMA connection channel resources to the bound device, the bound device re-establishes the request to the available network card.

[0111] Example 6

[0112] On the basis of Example 4 or Example 5, Figure 6 As shown, the failure of the network device is indexed to the resources of the corresponding storage protocol layer, and the connection of the storage protocol layer is closed to make the client aware of it, which specifically includes the following sub-steps:

[0113] S41, if it is a TCP network card device, when a single path fails, use the Linux Bond function to switch to an available network card device;

[0114] S42, when the entire Bond device fails, all iSCSI connections corresponding to the device are judged, and the corresponding iSCSI connection completion callback functions are called respectively; after passing through the storage protocol and network layer fault handling modules, the iSCSI initiator client will receive an error code of an unknown error on the server side, which will trigger a reconnection request; finally, after reconnecting to the target again, the socket connection or RDMA connection channel resources are re-established to the bound device, and the bound device re-establishes the request to the available network card.

[0115] Example 7

[0116] A computer device includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in Example 1 is executed.

[0117] Example 8

[0118] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in any one of Embodiments 2 to 3 is executed.

[0119] Example 9

[0120] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in any one of Embodiments 4 to 5 is executed.

[0121] Example 10

[0122] A computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method described in Example 6 is executed.

[0123] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0124] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0125] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. A storage protocol and network layer fault handling module, characterized in that: include: A new fault registration management module is added to the Linux kernel. This module is used to record iSCSI connections and iSER connections, and also records the related callback functions of the upper layer. When a problem occurs in Linux Bond, the callback function recorded in the fault registration management module is called according to the triggered error event to complete the fault processing.

2. A storage protocol and network layer fault handling method, characterized in that: The steps include: S1, add an error handling callback function linked list in the Linux kernel, the elements in the linked list are used to record: iSCSI connection resources of each storage protocol layer, completion callback functions related to the connection resources, related fault activity events, and unique identifiers of different resource devices under the bound device and device-related callback functions; S2, when the Linux Bond device receives a link failure of a single network card or the entire bonded device, different activity events will occur, and the callback function of the network device will be triggered according to the type of the corresponding activity event.

3. The storage protocol and network layer fault handling method according to claim 2, characterized in that: In step S2, triggering the callback function of the network device according to the type of the corresponding activity event specifically includes the following sub-steps: S21, in the callback function of the device, determine the corresponding iSCSI connection resource according to the resource ID of the network resource device, and then access the completion callback function corresponding to the connection resource; S22, in the completion callback function of the connection resource, calling the connection closing function to close the iSCSI connection; S23, after the iSCSI initiator finds that the connection of the target end is disconnected, it uses the original mechanism of its kernel to trigger the re-login connection reconnection mechanism to re-establish a new connection.

4. A storage protocol and network layer fault perception method, characterized in that: Establishing a connection relationship between the storage protocol layer and the network device layer based on the storage protocol and network layer fault processing module described in claim 1; The network device failure is then indexed to the corresponding storage protocol layer resources, and the connection at the storage protocol layer is closed to make the client aware of it.

5. The storage protocol and network layer fault perception method according to claim 4, characterized in that: The method of indexing the failure of the network device to the corresponding storage protocol layer resources and closing the connection of the storage protocol layer so as to make the client aware of it specifically includes the following sub-steps: S31, if it is an RDMA network card device, when a single path is disconnected, the Linux Bond device is considered unusable, which triggers the failure of the entire network card device; S32, determining all iSCSI connections corresponding to the device, and calling their corresponding iSCSI connection completion callback functions respectively; S33, after passing through the storage protocol and network layer fault processing modules, the iSCSI initiator client will receive an error code of an unknown error from the server, which will trigger a reconnection request; S34, after reconnecting to the target end, re-establishing the socket connection or RDMA connection channel resources to the bound device, the bound device re-establishes the request to the available network card.

6. The storage protocol and network layer fault perception method according to any one of claims 4 or 5, characterized in that: The method of indexing the failure of the network device to the corresponding storage protocol layer resources and closing the connection of the storage protocol layer so as to make the client aware of it specifically includes the following sub-steps: S41, if it is a TCP network card device, when a single path fails, use the Linux Bond function to switch to an available network card device; S42, when the entire Bond device fails, all iSCSI connections corresponding to the device are judged, and the corresponding iSCSI connection completion callback functions are called respectively; after passing through the storage protocol and network layer fault handling modules, the iSCSI initiator client will receive an error code of an unknown error on the server side, which will trigger a reconnection request; finally, after reconnecting to the target again, the socket connection or RDMA connection channel resources are re-established to the bound device, and the bound device re-establishes the request to the available network card.

7. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to claim 1 is executed.

8. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to any one of claims 2 to 3 is executed.

9. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to any one of claims 4 to 5 is executed.

10. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to claim 6 is executed.

Images