Network card fault switching method, device, electronic device, storage medium and product

Abstract

The present application discloses a network card fault switching method, device, electronic device, storage medium and product, which relate to the field of computer network technology, including obtaining a large page memory pool pre-allocated for each network card; determining a target polling thread in response to receiving a connection request from a storage client; obtaining a target network card for establishing a connection in response to the target polling thread detecting a network card fault switching event, and applying for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that remote direct memory access resources need to be reinitialized when a network card fails over, resulting in connection interruption and business stagnation, and achieves the technical effect of using the large page memory pool of the target network card immediately after the network card fails over, without waiting for resource re-creation, accelerating connection establishment, and improving business continuity.

Description

Technical Field

[0001] The present application relates to the field of computer network technology, and in particular to a network card fault switching method, electronic equipment, storage medium, and product. Background Art

[0002] In the relevant network card failover solution, the establishment of remote direct memory access connections depends on the queue pair and memory registration of the physical network card. Once the primary network card fails, the remote direct memory access resources need to be reinitialized during the network card failover, resulting in connection interruption and business stagnation. Summary of the Invention

[0003] The present application provides a network card fault switching method, device, electronic device, storage medium and product to at least solve the problem in the related art that when a network card fault is switched, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation.

[0004] This application provides a network card failover method, including:

[0005] Get the large page memory pool pre-allocated for each network card;

[0006] In response to receiving a connection request from a storage client, determining a target polling thread;

[0007] In response to the target polling thread detecting a network card failover event, the target network card for establishing a connection is obtained, and huge page memory is requested from a huge page memory pool pre-allocated by the target network card. The huge page memory is used to initialize remote direct memory access resources.

[0008] The present application also provides a network card failure switching device, comprising:

[0009] An acquisition unit is used to obtain the large page memory pool pre-allocated by each network card;

[0010] a determining unit, configured to determine a target polling thread in response to receiving a connection request from a storage client;

[0011] The application unit is used to obtain the target network card for establishing a connection in response to the target polling thread monitoring the network card failure switching event, and apply for huge page memory from the huge page memory pool pre-allocated by the target network card. The huge page memory is used to initialize remote direct memory access resources.

[0012] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned network card failure switching methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned network card failure switching methods are implemented.

[0014] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned network card failure switching methods when executed by a processor.

[0015] Through this application, the present application discloses a network card fault switching method, device, electronic device, storage medium and product, which obtains a large page memory pool pre-allocated for each network card; determines a target polling thread in response to receiving a connection request from a storage client; in response to the target polling thread monitoring a network card fault switching event, obtains the target network card for establishing a connection, and applies for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that when a network card fails over, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation, and achieves the technical effect of using the large page memory pool of the target network card immediately after the network card fails over, without waiting for resources to be re-created, speeding up connection establishment and improving business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A schematic diagram of the structure of a distributed storage system based on RDMA dual network cards provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a network card fault switching method provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a network card failover structure of a distributed storage system based on RDMA dual network cards provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of a method implemented by a storage service client in a distributed storage system provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of a network card fault switching device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] In order to facilitate those skilled in the art to better understand the technical solutions described in the embodiments of the present disclosure, the technical terms in the embodiments of the present disclosure are explained as follows before introducing the embodiments of the present disclosure.

[0025] Central Processing Unit (CPU): The core component of a computer, responsible for executing instruction sets and controlling the operation of the entire system.

[0026] Remote Direct Memory Access (RDMA): A high-performance network communication technology that allows one machine on a network to directly read and write the memory of another machine without excessive intervention from the operating system and CPU.

[0027] iSCSI Extensions for RDMA (iSER): used to efficiently transfer block storage data.

[0028] Linux system network bonding technology (Bond): allows multiple physical network cards to be combined into a logical interface to achieve redundancy or load balancing, and supports active / standby or load balancing modes for multiple network cards.

[0029] iSCSI storage service target (TGT): allows storage clients to connect to and access storage resources through the network.

[0030] Input / Output (IO): The process of exchanging data between a computer system and the outside world (such as users, storage devices, networks, etc.).

[0031] As data centers grow in demand for high-performance storage, distributed storage systems based on RDMA (Remote Direct Memory Access) are widely used due to their low latency and high throughput. RDMA bypasses the operating system kernel and directly accesses remote memory, eliminating the need for CPU access, significantly improving network communication efficiency. Currently, RDMA-based protocols such as iSER are widely used in distributed storage scenarios.

[0032] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a distributed storage system based on RDMA dual network cards provided in an embodiment of the present application. Taking a distributed storage cluster consisting of two storage nodes as an example, Node A and Node B each have two RDMA network cards managed through the Linux system's Bond technology. Network card 1 is the primary network card, and network card 2 is the backup network card. The iSCSI client and server-side TGT service are interconnected via the iSER protocol, and data communication is carried out between storage nodes A and B via an RDMA connection. The iSER protocol combines the traditional iSCSI protocol with RDMA for block storage data transmission between front-end clients and storage systems, supporting high-concurrency, low-latency IO operations.

[0033] However, RDMA's hardware dependencies (such as the network card queue pair (QP) and memory registration mechanism) make it challenging to recover from network card failures. Establishing an RDMA connection relies on the physical network card's queue pair and memory registration. If the primary network card fails, the QP must be reinitialized, resulting in connection interruption and service stagnation. Furthermore, iSER's session state is deeply bound to the hardware, requiring a complete context reconstruction after a failure. This complex recovery process and the complete interruption of inter-node communication during this period severely impact the IO service quality provided by the distributed storage cluster. These IO fluctuations are unacceptable for users in scenarios such as finance and real-time analytics.

[0034] The present application includes obtaining a large page memory pool pre-allocated for each network card; determining a target polling thread in response to receiving a connection request from a storage client; obtaining a target network card for establishing a connection in response to the target polling thread detecting a network card failover event, and applying for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that when a network card fails over, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation. The large page memory pool of the target network card is used immediately after the network card fails over, without waiting for resource re-creation, thereby accelerating connection establishment and improving business continuity.

[0035] The embodiment of the present disclosure provides a network card fault switching method, the execution subject of which can be a software module or service process running on a computer system, such as a network driver, a resource manager, etc., and can be applied to distributed storage systems and high-performance computing clusters.

[0036] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0037] Figure 2 A flowchart of a network card fault switching method provided by an embodiment of the present disclosure.

[0038] like Figure 2 As shown, the method comprises the following steps:

[0039] Step 201, obtaining the large page memory pool pre-allocated by each network card;

[0040] In some embodiments, the traditional memory page size is typically 4KB, while the large page memory can be 2MB or even 1GB. Using large page memory can reduce the number of Translation Lookaside Buffer (TLB) misses, thereby improving memory access speed.

[0041] In some embodiments, in order to optimize RDMA operations, a large page memory pool is usually pre-allocated to each network card for registering memory regions (MRs), queues, and other resources.

[0042] In some embodiments, each network card has an independent large page memory pool to avoid performance degradation caused by cross-node access. The memory pool can be pre-allocated during system startup or application initialization to reduce dynamic allocation overhead at runtime.

[0043] Step 202: In response to receiving a connection request from a storage client, determining a target polling thread;

[0044] In some embodiments, a storage client refers to an entity that initiates data read and write requests to a backend storage service. The storage client can be a virtual machine, a container application, a computing node, or a database / file system component.

[0045] In some embodiments, the polling thread is used to monitor network events, such as new connections, data arrival, completion events, and other working threads. The polling thread in this application takes the poller thread as an example.

[0046] In some embodiments, a polling thread with the least number of connections among multiple polling threads may be determined as a target polling thread. During a failover, this thread will detect a change in the network card status and trigger a recovery process.

[0047] Step 203: In response to the target polling thread detecting the network card failover event, the target network card for establishing the connection is obtained, and huge page memory is requested from the huge page memory pool pre-allocated by the target network card. The huge page memory is used to initialize remote direct memory access resources.

[0048] In some embodiments, a network card failure switching event refers to a process in which a primary network card fails and the system automatically switches to a backup network card. The primary network card failure may be a link disconnection, a driver abnormality, or other failure.

[0049] In some embodiments, the target network card refers to a new active network card after failover, and all new connections are bound to the network card and use its corresponding resources, such as a large page memory pool, an RDMA context, and the like.

[0050] In some embodiments, as Figure 3 As shown, Figure 3 A structural diagram of network card failure switching in a distributed storage system based on RDMA dual network cards is provided in an embodiment of the present application, in which the primary network card 1 of node A fails, and network card 2 is switched to the primary network card. The iSER connection between the iSCSI client and the TGT and the RDMA connection between the storage services on nodes A and B are switched from network card 1 to network card 2.

[0051] In some embodiments, the distributed storage system provides a redirection service to the outside world. When a client wants to connect to the storage system, it first obtains a connectable storage node through the redirection service. The storage cluster will select one of the nodes as a VIP node to provide the redirection service. The iSCSI client uses a multi-path approach to connect to the storage service, that is, a client will have multiple paths to the server, and each path will have a connection. The normal process is that each path needs to obtain a redirection message from the storage service to establish a connection. After obtaining the redirection information returned by the server, it will connect to the corresponding storage node. Therefore, the establishment of the connection between the client and the server requires all paths to be completed. When the RDMA network card fails and the primary and backup network cards are switched, the iSCSI client recognizes that the RDMA link is abnormal and needs to disconnect all paths and then obtain redirection information from the redirection service to establish a new connection. This process is time-consuming and affects the continuity of business IO. In response to the problems existing in the above process, the present application changes the need to obtain redirection information for the establishment of each path to only the first path to obtain redirection information from the redirection service, and the remaining paths directly reuse the redirection information of the first path. This can significantly shorten the time required for all links to establish connections and ensure the continuity of IO when the network card fails.

[0052] In some embodiments, remote direct memory access resources may include queue pairs, memory regions, completion queues, protection domains, and other resources, which need to be reinitialized after a network card switch to support new connections.

[0053] In some embodiments, when the distributed storage system detects a network card failover event, the target polling thread perceives the event, determines the target network card, applies for memory from the large page memory pool corresponding to the network card, and uses the memory to initialize RDMA resources. Subsequent new connections will use these resources to establish communication, and old connections can be gradually closed and resources released back to the pool.

[0054] The present application includes obtaining a large page memory pool pre-allocated for each network card; determining a target polling thread in response to receiving a connection request from a storage client; obtaining a target network card for establishing a connection in response to the target polling thread detecting a network card failover event, and applying for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that when a network card fails over, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation. The large page memory pool of the target network card is used immediately after the network card fails over, without waiting for resource re-creation, thereby accelerating connection establishment and improving business continuity.

[0055] In some embodiments, obtaining a large page memory pool pre-allocated by each network card includes:

[0056] Obtain configuration information of the large page memory pool and a set of network cards associated with the first interface in the distributed storage system;

[0057] In some embodiments, the configuration information of the huge page memory pool can be obtained by parsing the configuration file of the storage service. The configuration information may include the type and size of the huge page memory. The configuration information of the huge page memory pool can specify a set of parameters for how to divide the huge page memory, such as how many huge pages are allocated to each network card, the huge page size (such as 2MB, 1GB), the type of huge page memory, and whether the memory pool can be dynamically expanded.

[0058] In some embodiments, the size of the memory pool can be dynamically adjusted according to actual business pressure.

[0059] In some embodiments, a distributed storage system refers to a storage architecture in which data is distributed across multiple nodes and accessed through a network.

[0060] In some embodiments, the first interface refers to the primary network interface of the node in the distributed storage system for external services, which is usually a logical interface, such as a bond interface. The network card set refers to a list of one or more physical network cards associated with the first interface, which can be used for scenarios such as active-standby redundancy, load balancing, or multipath RDMA, as mentioned above. Figure 3 The network card sets in are network card 1 and network card 2.

[0061] Based on the huge page memory pool configuration information, determine the huge page memory corresponding to each network card in the network card set;

[0062] In some embodiments, based on the large page memory pool size and type in the large page memory pool configuration information, a large page memory of a corresponding size is requested for each network card entity.

[0063] The huge page memory corresponding to each network card is pooled to obtain a pre-allocated huge page memory pool for each network card.

[0064] In some embodiments, by pooling the large page memory corresponding to each network card, the large page memory is divided into fixed-size blocks, and the free list and allocation / release logic are maintained to achieve efficient memory management and avoid fragmentation.

[0065] In some embodiments, the network card entity information associated with the system bond interface is obtained, and based on the huge page memory pool size and type parsed in the previous step, huge page memory of corresponding size is applied for each network card entity, and pooled management is performed based on the type. In this way, each poller thread is associated with the primary and backup network cards under the bond interface, and each network card is associated with its own exclusive huge page memory pool.

[0066] In some embodiments, by obtaining the network card set and large page memory pool configuration information associated with the first interface, exclusive large page memory resources are pre-allocated and pooled for each network card, thereby achieving memory localization, improving RDMA performance and system fault tolerance.

[0067] In some embodiments, obtaining configuration information of the large page memory pool includes:

[0068] In response to a storage service being started in the distributed storage system, obtaining a configuration file of the storage service;

[0069] In some embodiments, the storage service refers to a service program running on each storage node, which is responsible for receiving storage client requests and processing data reading and writing; the storage service configuration file refers to a text or JSON / YAML format configuration file loaded when the storage service is started, which includes network parameters, number of threads, memory pool size, RDMA configuration, etc.

[0070] Parse the configuration file to determine a polling thread set, where the polling thread set includes at least two polling threads;

[0071] In response to the polling thread being started, configuration information of the large page memory pool is determined from a configuration file of the storage service.

[0072] In some embodiments, when the distributed storage service is started, the configuration file is loaded, the configuration content is parsed, thread-related configuration items and memory-related configuration items are extracted, multiple Poller threads are started, and they are bound to different network cards or event loops respectively. According to the configuration, an exclusive large page memory pool is pre-allocated for each network card for subsequent rapid initialization of RDMA resources (MR, QP).

[0073] In some embodiments, when the storage service is started on each storage service node, the TGT module first parses the number of poller threads based on the configuration file. Poller threads are primarily responsible for resource management of iSCSI client connections and the sending and receiving of data and events. After starting, the poller threads parse the configuration file to obtain the type and size of their respective huge page memory pools, and can dynamically adjust the memory pool size based on actual business pressure.

[0074] In some embodiments, by loading the configuration file during the storage service startup phase, dynamically building a polling thread set and initializing the large page memory pool, flexible resource configuration and high-performance network communication support are achieved, which is suitable for distributed storage systems with multiple network cards and RDMA acceleration.

[0075] In some embodiments, after pooling the huge page memory corresponding to each network card to obtain a huge page memory pool pre-allocated to each network card, the network card failover method further includes:

[0076] A corresponding remote direct memory access resource is created for the polling thread, where the remote direct memory access resource includes at least one of a protection domain and a completion queue.

[0077] In some embodiments, a protection domain (PD) is a resource isolation mechanism in RDMA. All registered memory regions (MRs) and queue pairs (QPs) must belong to a PD. The protection domain is used to implement memory access permission control to prevent illegal access to the memory of other processes or devices.

[0078] In some embodiments, a completion queue (CQ) is a data structure in RDMA used to receive operation completion events. After each work request (WR) is completed, a completion event is generated and placed in the corresponding CQ. The polling thread periodically checks and processes these events.

[0079] In some embodiments, after creating corresponding remote direct memory access resources for the polling thread, resources such as PD and CQ are saved in a thread-private structure to reduce cross-thread access overhead and improve concurrency performance.

[0080] In some embodiments, by binding exclusive RDMA resources (such as protection domains, completion queues, etc.) to each polling thread, resource isolation between threads and an efficient event processing mechanism are achieved, which is suitable for multi-threaded RDMA network communication and distributed storage systems.

[0081] In some embodiments, in response to receiving a connection request from a storage client, determining a target polling thread includes:

[0082] In response to receiving a connection request from a storage client, obtaining a resource occupancy value and a business pressure value of each polling thread in the polling thread set;

[0083] In some embodiments, the resource usage value indicates the resource usage of the thread pool or the bound network card / device, including but not limited to CPU usage, memory usage, network bandwidth, queue depth, etc. The business pressure value indicates the intensity of the business load currently processed by the thread, including but not limited to the current number of connections, number of unprocessed events, request queue length, and latency indicators.

[0084] Based on the resource occupancy value and the business pressure value, a target polling thread is determined from the polling thread set.

[0085] In some embodiments, the load score of each thread is comprehensively calculated. The lower the score, the more idle it is. The optimal thread is selected as the target thread, and the connection is handed over to the thread for processing. The thread is the target polling thread.

[0086] In some embodiments, based on the resource occupancy value and the service pressure value, a target polling thread is determined from a set of polling threads, and the network card failover method further includes updating a load status of the thread.

[0087] In some embodiments, by dynamically collecting the resource occupancy and business pressure indicators of the polling thread when a connection request is received and selecting the most appropriate target thread accordingly, efficient load balancing and resource scheduling are achieved, which is suitable for multi-threaded, high-concurrency distributed storage systems.

[0088] In some embodiments, in response to the target polling thread detecting a network card failover event, obtaining the target network card for establishing a connection, and applying for huge page memory from the huge page memory pool pre-allocated by the target network card, the network card failover method further includes:

[0089] Close the connection associated with the network card in standby mode;

[0090] In some embodiments, a network card in standby mode refers to a physical or virtual network card that is not currently used for data transmission and is used only as a failover backup. When the primary network card fails, the standby network card will be promoted to continue providing services as the primary network card.

[0091] In some embodiments, a connection represents a communication channel between a client and a server, and is typically represented by a queue pair in RDMA.

[0092] Put the huge page memory corresponding to the connection back into the huge page memory pool pre-allocated by the network card, and release the queue pair resources requested for the connection.

[0093] In some embodiments, when a network card fails over, the newly established connection will obtain the information of the network card where the connection is located, then apply for memory from the associated network card memory pool and initialize related resources. The failed network card switches to Slave mode (standby mode), and the network card associated with the new connection becomes Active mode. When the poller thread detects that the master and slave network cards have switched, it will actively close all connections established under the Slave mode network card. After the connection is closed, all memory resources will be recycled and put back into the free memory pool for subsequent use. At the same time, the QP resources applied for each connection will be destroyed. The poller thread associated with RDMA resources such as PD and CQ does not need to be recycled. When the Slave mode network card switches to Active mode again, the RDMA resources such as PD and CQ will be directly reused, which can speed up the connection.

[0094] In some embodiments, when the system detects a network card switching event and the original primary network card becomes a standby network card, it traverses all connections associated with the standby network card, closes these connections, cleans up the RDMA queue pairs (QPs), terminates ongoing data transfers, reclaims resources used by the connection, and returns huge page memory to the corresponding huge page memory pool. This releases RDMA resources such as QPs and MRs, updates the connection status table and resource usage statistics, and reserves resources for subsequent new connections to support seamless switching.

[0095] In some embodiments, as Figure 4 As shown, Figure 4 A flow chart of a method implemented by a storage service client in a distributed storage system provided in an embodiment of the present application, wherein the step of large page memory pooling is the step of initializing the polling thread. After the initialization is completed, wait for the connection of the storage client. If there is a connection, the corresponding polling thread is allocated to the connection.

[0096] In some embodiments, by actively closing the connections carried by the network card and reclaiming the large page memory and RDMA queue resources it uses when the network card enters the standby mode, efficient resource reuse and system fault tolerance are achieved, which is suitable for multi-path RDMA and distributed storage systems.

[0097] In some embodiments, the network card failover method further includes:

[0098] Initialize the connection information corresponding to each node in the distributed storage system, where the connection information includes at least a status bit;

[0099] In some embodiments, a node refers to an independent computing unit in the system, typically corresponding to a server or container instance, and each node may establish multiple connections with other nodes to achieve data transmission and synchronization.

[0100] In some embodiments, the connection information in the embodiments of the present application takes the connection structure as an example, which is a data structure used to describe the communication status between two nodes, recording the information and status of the forwarding node. The connection information may include but is not limited to the current connection status (active / disconnected / migrated), sending thread status, receiving thread status, last heartbeat timestamp, and fault counter.

[0101] In some embodiments, the status bit is used to indicate the status of the current connection, including at least normal connection and unavailable connection.

[0102] In response to the target polling thread detecting a network card failover event, determining a sending thread state and a receiving thread state;

[0103] Based on the sending thread state and the receiving thread state, the state bit in the connection information corresponding to the node is determined.

[0104] In some embodiments, when a network card failover occurs, the sending thread fails to send data and marks the status bit in the corresponding connection structure as dead.

[0105] In some embodiments, when a network card failover occurs, the receiving thread detects an exception in the corresponding file descriptor fd through the epoll_wait interface and marks the status bit in the connection information corresponding to the node, such as the status bit in the connection structure, as dead.

[0106] In some embodiments, by initializing connection information for each node and dynamically evaluating the status of the sending and receiving threads when the network card fails over, the connection status bits are accurately updated, thereby achieving refined management of the communication status and high availability guarantee of the distributed storage system.

[0107] In some embodiments, after determining the status bit in the connection information corresponding to the node based on the sending thread status and the receiving thread status, the network card failover method further includes:

[0108] In response to receiving an input / output request, obtaining a status bit in the connection information corresponding to the node;

[0109] In some embodiments, input and output requests refer to read and write operations initiated by a client or within the system. In a distributed storage system, they typically include data block read requests, data block write requests, and metadata operations.

[0110] If the status bit in the connection information corresponding to the node is unavailable, the input and output request will be added to the request forwarding failure queue, and a timestamp identifier will be added to the connection information corresponding to the node.

[0111] In some embodiments, the request forwarding failure queue is used to store I / O requests that cannot be processed immediately due to unreachable target node connections.

[0112] In some embodiments, the timestamp identifier is used to record the earliest occurrence time of connection unavailability, which can be used for subsequent timeout determination or priority sorting.

[0113] In some embodiments, when forwarding completed IO requests between storage nodes, the corresponding connection structure is first obtained. If the status bit of the connection is found to be dead or the data transmission fails, the IO request is placed in the forwarding failure queue for caching, and the IO request is timestamped.

[0114] In some embodiments, by checking the connection status of the target node when receiving an IO request, if the connection is unavailable, it is added to the failure queue and a timestamp is marked in the connection information, thereby implementing a request caching and subsequent retry mechanism for unreachable nodes, thereby improving the system's fault tolerance and stability.

[0115] In some embodiments, if the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to the request forwarding failure queue, and after adding a timestamp identifier to the connection information corresponding to the node, the network card failover method further includes:

[0116] Use the reconnection mechanism to reestablish the connection with the storage client;

[0117] In response to a successful connection with the storage client, a status bit in the connection information corresponding to the node is updated.

[0118] In some embodiments, the reconnection mechanism is a mechanism that automatically attempts to reestablish a communication channel after a network connection is interrupted, and is commonly used in protocol stacks such as TCP and RDMA.

[0119] In some embodiments, when a connection is detected to be unavailable or a reconnection is actively triggered, reconnection logic such as RDMA QP reconstruction is executed. If the connection is successfully established, the status bit in the connection information is updated, and the previous timestamp, error counter and other information are cleared.

[0120] In some embodiments, by starting a reconnection mechanism after a connection is disconnected and promptly updating the connection status bit after the connection is restored, automatic recovery and status synchronization of network connections in a distributed storage system are achieved, thereby improving the system's fault tolerance and availability.

[0121] In some embodiments, if the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to the request forwarding failure queue, and after adding a timestamp identifier to the connection information corresponding to the node, the network card failover method further includes:

[0122] Get the timestamp corresponding to each request in the request forwarding failure queue;

[0123] Based on the timestamp corresponding to each request, a first request is determined and discarded, and the first request is a timed-out request.

[0124] In some embodiments, the timestamp corresponding to each request is used in this scenario to record the time when the request joins the failure queue.

[0125] In some embodiments, the first request refers to a request that has been in the failure queue for longer than a set threshold. Such requests should be actively discarded to avoid resource occupation.

[0126] In some embodiments, discarding the first request refers to removing the first request from the failure queue and releasing the resources occupied by the first request, and at the same time notifying the upper layer application that the request has failed.

[0127] In some embodiments, all requests in the request forwarding failure queue are traversed, the timestamp information of each request is obtained, and the difference between the current time and the request entry time is calculated. If the difference is greater than the set timeout threshold, such as 60 seconds, the corresponding request is discarded.

[0128] In some embodiments, by setting a timestamp for each request in the request forwarding failure queue and judging whether it has timed out based on the difference between the current time and the time of entry into the queue, the timed out requests can be identified and automatically discarded, thereby effectively controlling memory usage and improving the robustness and resource utilization of the system.

[0129] In some embodiments, after obtaining the timestamp corresponding to each request in the request forwarding failure queue, the network card failover method further includes:

[0130] Determine, based on a timestamp corresponding to each request, a second request, where the second request is a non-timed-out request;

[0131] Determine a status bit in the connection information corresponding to the second request;

[0132] In response to the status bit in the connection information corresponding to the second request being available, the second request is forwarded and deleted from the request forwarding failure queue.

[0133] In some embodiments, the second request refers to an I / O request in the request forwarding failure queue that has not exceeded the set timeout period, indicating that the request may be successfully forwarded because the target node has recovered.

[0134] In some embodiments, all requests in the failure queue are traversed, and whether they have timed out is determined based on the timestamp. If they have not timed out, they are marked as "second request", and the target node connection information corresponding to the request is obtained. It is checked whether the connection status bit is in the "available" state. If the connection is available, an attempt is made to forward the request. If the forwarding is successful, the request is deleted from the failure queue and its resources are released.

[0135] In some embodiments, by performing timestamp filtering and connection status judgment on requests in the failure queue, requests that have not timed out are automatically retried after the connection is restored and deleted from the failure queue, thereby achieving intelligent recovery and resource recovery of abnormal requests in the distributed storage system.

[0136] The present application includes obtaining a large page memory pool pre-allocated for each network card; determining a target polling thread in response to receiving a connection request from a storage client; obtaining a target network card for establishing a connection in response to the target polling thread detecting a network card failover event, and applying for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that when a network card fails over, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation. The large page memory pool of the target network card is used immediately after the network card fails over, without waiting for resource re-creation, thereby accelerating connection establishment and improving business continuity.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0138] The embodiment of the present application further provides a network card failure switching device 500, Figure 5 A schematic diagram of a network card failure switching device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 Shown, including:

[0139] An acquisition unit 501 acquires a large page memory pool pre-allocated by each network card;

[0140] A determining unit 502 is configured to determine a target polling thread in response to receiving a connection request from a storage client;

[0141] The application unit 503 is used to obtain the target network card for establishing a connection in response to the target polling thread monitoring the network card failover event, and apply for huge page memory from the huge page memory pool pre-allocated by the target network card. The huge page memory is used to initialize remote direct memory access resources.

[0142] Furthermore, in a possible implementation of the embodiment of the present disclosure, the acquiring unit 501 is configured to:

[0143] Obtain configuration information of the large page memory pool and a set of network cards associated with the first interface in the distributed storage system;

[0144] Based on the huge page memory pool configuration information, determine the huge page memory corresponding to each network card in the network card set;

[0145] The huge page memory corresponding to each network card is pooled to obtain a pre-allocated huge page memory pool for each network card.

[0146] Furthermore, in a possible implementation of the embodiment of the present disclosure, the acquiring unit 501 is configured to:

[0147] In response to a storage service being started in the distributed storage system, obtaining a configuration file of the storage service;

[0148] Parse the configuration file to determine a polling thread set, where the polling thread set includes at least two polling threads;

[0149] In response to the polling thread being started, configuration information of the large page memory pool is determined from a configuration file of the storage service.

[0150] Furthermore, in a possible implementation of the embodiment of the present disclosure, the network card failure switching device 500 further includes a resource creation unit, which is configured to:

[0151] A corresponding remote direct memory access resource is created for the polling thread, where the remote direct memory access resource includes at least one of a protection domain and a completion queue.

[0152] Furthermore, in a possible implementation of the embodiment of the present disclosure, the determining unit 502 is configured to:

[0153] In response to receiving a connection request from a storage client, obtaining a resource occupancy value and a business pressure value of each polling thread in the polling thread set;

[0154] Based on the resource occupancy value and the business pressure value, a target polling thread is determined from the polling thread set.

[0155] Furthermore, in a possible implementation of the embodiment of the present disclosure, the network card failure switching device 500 further includes a resource releasing unit, which is configured to:

[0156] Close the connection associated with the network card in standby mode;

[0157] Put the huge page memory corresponding to the connection back into the huge page memory pool pre-allocated by the network card, and release the queue pair resources requested for the connection.

[0158] Furthermore, in a possible implementation of the embodiment of the present disclosure, the network card failure switching device 500 further includes a status bit determination unit, which is configured to:

[0159] Initialize the connection information corresponding to each node in the distributed storage system, where the connection information includes at least a status bit;

[0160] In response to the target polling thread detecting a network card failover event, determining a sending thread state and a receiving thread state;

[0161] Based on the sending thread state and the receiving thread state, the state bit in the connection information corresponding to the node is determined.

[0162] Furthermore, in a possible implementation of the embodiment of the present disclosure, the network card failure switching device 500 further includes an adding unit, which is configured to:

[0163] In response to receiving an input / output request, obtaining a status bit in the connection information corresponding to the node;

[0164] If the status bit in the connection information corresponding to the node is unavailable, the input and output request will be added to the request forwarding failure queue, and a timestamp identifier will be added to the connection information corresponding to the node.

[0165] Furthermore, in a possible implementation of the embodiment of the present disclosure, the network card failure switching device 500 further includes an updating unit, which is configured to:

[0166] Use the reconnection mechanism to reestablish the connection with the storage client;

[0167] In response to a successful connection with the storage client, a status bit in the connection information corresponding to the node is updated.

[0168] Furthermore, in a possible implementation of the embodiment of the present disclosure, the determining unit 502 is configured to:

[0169] Get the timestamp corresponding to each request in the request forwarding failure queue;

[0170] Based on the timestamp corresponding to each request, a first request is determined and discarded, and the first request is a timed-out request.

[0171] Furthermore, in a possible implementation of the embodiment of the present disclosure, the determining unit 502 is configured to:

[0172] Determine, based on a timestamp corresponding to each request, a second request, where the second request is a non-timed-out request;

[0173] Determine a status bit in the connection information corresponding to the second request;

[0174] In response to the status bit in the connection information corresponding to the second request being available, the second request is forwarded and deleted from the request forwarding failure queue.

[0175] The present application includes obtaining a large page memory pool pre-allocated for each network card; determining a target polling thread in response to receiving a connection request from a storage client; obtaining a target network card for establishing a connection in response to the target polling thread detecting a network card failover event, and applying for large page memory from the large page memory pool pre-allocated by the target network card. The large page memory is used to initialize remote direct memory access resources, which solves the technical problem in related solutions that when a network card fails over, remote direct memory access resources need to be reinitialized, resulting in connection interruption and business stagnation. The large page memory pool of the target network card is used immediately after the network card fails over, without waiting for resource re-creation, thereby accelerating connection establishment and improving business continuity.

[0176] For the description of the features in the embodiment corresponding to the network card failure switching device, please refer to the relevant description of the embodiment corresponding to the network card failure switching method, and no further details will be given here.

[0177] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned network card failover method embodiments.

[0178] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned network card failover method embodiments when running.

[0179] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0180] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned network card failure switching method embodiments are implemented.

[0181] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned network card failure switching method embodiments are implemented.

[0182] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] The above is a detailed introduction to a network card fault switching method, device, electronic device, storage medium and product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A network card fault switching method, characterized in that: include: In response to a storage service being started in the distributed storage system, obtaining a configuration file of the storage service, the storage service being a service program running on each storage node; Parsing the configuration file to determine a polling thread set, where the polling thread set includes at least two polling threads; In response to the polling thread being started, determining configuration information of a large page memory pool from a configuration file of the storage service; Obtaining a network card set associated with a first interface in the distributed storage system, where the first interface is a bond interface for external services provided by a node in the distributed storage system; Determine the huge page memory corresponding to each network card in the network card set based on the configuration information of the huge page memory pool; Pooling the huge page memory corresponding to each network card to obtain a huge page memory pool pre-allocated to each network card; Creating an exclusive remote direct memory access resource for the polling thread, wherein the exclusive remote direct memory access resource includes at least one of a protection domain and a completion queue; In response to receiving a connection request from a storage client, determining a target polling thread; In response to the target polling thread detecting a network card failover event, obtaining a target network card for establishing a connection, and applying for a huge page memory from a huge page memory pool pre-allocated by the target network card, wherein the huge page memory is used to initialize a remote direct memory access resource corresponding to the target network card; Close the connection associated with the network card in standby mode; Put the huge page memory corresponding to the connection associated with the network card in standby mode back to the huge page memory pool pre-allocated by the network card in standby mode, and release the queue pair resources applied for the connection associated with the network card in standby mode.

2. The network card failure switching method according to claim 1, wherein: In response to receiving a connection request from the storage client, determining a target polling thread includes: In response to receiving a connection request from a storage client, obtaining a resource occupancy value and a business pressure value of each polling thread in the polling thread set; Based on the resource occupancy value and the service pressure value, a target polling thread is determined from the polling thread set.

3. The network card failure switching method according to claim 1, wherein: The method further comprises: Initialize connection information corresponding to each node in the distributed storage system, wherein the connection information includes at least a status bit; In response to the target polling thread detecting a network card failover event, determining a sending thread state and a receiving thread state; Based on the sending thread state and the receiving thread state, a state bit in the connection information corresponding to the node is determined.

4. The network card failure switching method according to claim 3, characterized in that: After determining the status bit in the connection information corresponding to the node based on the sending thread status and the receiving thread status, the method further includes: In response to receiving an input / output request, obtaining a status bit in the connection information corresponding to the node; If the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to a request forwarding failure queue, and a timestamp identifier is added to the connection information corresponding to the node.

5. The network card failure switching method according to claim 4, characterized in that: If the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to a request forwarding failure queue, and a timestamp identifier is added to the connection information corresponding to the node. The method further includes: Re-establishing a connection with the storage client using a reconnection mechanism; In response to a successful connection with the storage client, a status bit in the connection information corresponding to the node is updated.

6. The network card failure switching method according to claim 5, characterized in that: If the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to a request forwarding failure queue, and a timestamp is added to the connection information corresponding to the node, the method further includes: Obtaining the timestamp corresponding to each request in the request forwarding failure queue; Based on the timestamp corresponding to each request, a first request is determined and discarded, where the first request is a timed-out request.

7. The network card failure switching method according to claim 6, wherein: After obtaining the timestamp corresponding to each request in the request forwarding failure queue, the method further includes: Determine a second request based on the timestamp corresponding to each request, where the second request is a non-timed-out request; Determine a status bit in the connection information corresponding to the second request; In response to the status bit in the connection information corresponding to the second request being available, the second request is forwarded and deleted from the request forwarding failure queue.

8. A network card failure switching device, characterized in that: include: a first acquiring unit, configured to acquire a configuration file of a storage service in response to startup of the storage service in the distributed storage system, wherein the storage service is a service program running on each storage node; a first determining unit, configured to parse the configuration file and determine a polling thread set, wherein the polling thread set includes at least two polling threads; A second determining unit is configured to determine configuration information of a large page memory pool from a configuration file of the storage service in response to the polling thread being started; A second acquiring unit is configured to acquire a set of network cards associated with a first interface in the distributed storage system, where the first interface is a bond interface for external services provided by a node in the distributed storage system; A third determining unit is configured to determine the huge page memory corresponding to each network card in the network card set based on the configuration information of the huge page memory pool; A pooling unit, configured to perform pooling processing on the huge page memory corresponding to each network card to obtain a huge page memory pool pre-allocated to each network card; A creating unit, configured to create an exclusive remote direct memory access resource for the polling thread, wherein the exclusive remote direct memory access resource includes at least one of a protection domain and a completion queue; a fourth determining unit, configured to determine a target polling thread in response to receiving a connection request from a storage client; an application unit, configured to, in response to the target polling thread detecting a network card failover event, obtain a target network card for establishing a connection, and apply for a huge page memory from a huge page memory pool pre-allocated by the target network card, wherein the huge page memory is used to initialize a remote direct memory access resource corresponding to the target network card; A shutdown unit, configured to shut down a connection associated with the network card in standby mode; A releasing unit is used to put the huge page memory corresponding to the connection associated with the network card in the standby mode back to the huge page memory pool pre-allocated by the network card in the standby mode, and release the queue pair resources applied for the connection associated with the network card in the standby mode.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the network card failover method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the network card failure switching method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the network card failure switching method according to any one of claims 1 to 7 are implemented.

Images