Network card fault switching method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a network card fault switching method and device, electronic equipment, a storage medium and a product, and relates to the technical field of computer networks, and the method comprises the steps: obtaining a pre-allocated large-page memory pool of each network card; in response to a received connection request of the storage client, determining a target polling thread; and in response to a network card failover event monitored by the target polling thread, obtaining the target network card which establishes the connection, and applying for a large-page memory from a large-page memory pool pre-allocated to the target network card, the large-page memory being used for initializing remote direct memory access resources, thereby solving the problem that in a related scheme, when network card failover occurs, the remote direct memory access resources cannot be accessed to the target network card, and the remote direct memory access resources cannot be accessed to the target network card. The technical problem that connection interruption and service stagnation are caused due to the fact that remote direct memory access resources need to be initialized again is solved, and the technical effects that a large-page memory pool of a target network card is immediately used after network card fault switching, waiting for resource recreation is not needed, connection establishment is accelerated, and service continuity is improved are achieved.

Description

Technical Field

[0001] This application relates to the field of computer network technologies, and in particular, to a network card failure switching method, an electronic device, a storage medium, and a product. Background Art

[0002] In related network card failure switching solutions, the establishment of a Remote Direct Memory Access (RDMA) connection depends on the queue pairs and memory registration of physical network cards. Once the primary network card fails, during network card failure switching, the RDMA resources need to be re-initialized, resulting in connection interruption and service suspension. Summary of the Invention

[0003] This application provides a network card failure switching method, apparatus, electronic device, storage medium, and product to at least solve the problem in related technologies that during network card failure switching, the RDMA resources need to be re-initialized, resulting in connection interruption and service suspension.

[0004] This application provides a network card failure switching method, including: Obtaining a large page memory pool pre-allocated for each network card; Responding to a connection request received from a storage client to determine a target polling thread; Responding to a network card failure switching event detected by the target polling thread, obtaining a target network card for establishing a connection, and applying for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize the RDMA resources.

[0005] This application further provides a network card failure switching apparatus, including: An obtaining unit, configured to obtain a large page memory pool pre-allocated for each network card; A determining unit, configured to respond to a connection request received from a storage client to determine a target polling thread; An applying unit, configured to respond to a network card failure switching event detected by the target polling thread, obtain a target network card for establishing a connection, and apply for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize the RDMA resources.

[0006] This application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above network card failure switching methods when executing the computer program.

[0007] This application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above network card failure switching methods are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above network card failure switching methods when executed by a processor.

[0009] Through the present application, the present application discloses a network card failure switching method, device, electronic device, storage medium and product, which obtains a large page memory pool pre-allocated for each network card; in response to receiving a connection request from a storage client, determines a target polling thread; in response to the target polling thread detecting a network card failure switching event, obtains a target network card for establishing a connection, and applies for large page memory from the large page memory pool pre-allocated for the target network card, and the large page memory is used to initialize remote direct memory access resources, solving the technical problem that in the related solution, when the network card fails to switch, the remote direct memory access resources need to be re-initialized, resulting in connection interruption and service stagnation, and achieving the technical effect of immediately using the large page memory pool of the target network card after the network card fails to switch, without waiting for the resources to be re-created, accelerating the establishment of the connection, and improving the service continuity. Description of the Drawings

[0010] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0011] Figure 1 It is a schematic structural diagram of a distributed storage system based on RDMA dual network cards provided by an embodiment of the present application; Figure 2 It is a schematic flowchart of a network card failure switching method provided by an embodiment of the present application; Figure 3 It is a schematic structural diagram of network card failure switching of a distributed storage system based on RDMA dual network cards provided by an embodiment of the present application; Figure 4 It is a schematic flowchart of a method implemented by a storage server in a distributed storage system provided by an embodiment of the present application; Figure 5 It is a schematic structural diagram of a network card failure switching device provided by an embodiment of the present application. Detailed Embodiments

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0013] It should be noted that in the description of this application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0014] To facilitate better understanding of the technical solutions described in the embodiments of the present disclosure by those skilled in the art, before introducing the embodiments of the present disclosure, the following explanations are made for the technical terms in the embodiments of the present disclosure.

[0015] Central Processing Unit (CPU): The core component of a computer, responsible for executing instruction sets and controlling the operation of the entire system.

[0016] Remote Direct Memory Access (RDMA): A high-performance network communication technology that allows a machine in a network to directly read and write the memory of another machine without excessive intervention from the operating system and CPU.

[0017] iSCSI Extensions for RDMA (iSER): Used for efficient transmission of block storage data.

[0018] Linux system network bonding technology (Bond): Allows multiple physical network cards to be combined into a logical interface to achieve redundancy or load balancing, and supports the primary / backup or load balancing modes of multiple network cards.

[0019] iSCSI storage server (Target, TGT): Allows storage clients to connect to and access storage resources through the network.

[0020] Input / Output (IO): The process of data exchange between a computer system and the external world (such as users, storage devices, networks, etc.).

[0021] With the growing demand for high-performance storage in data centers, distributed storage systems based on RDMA (Remote Direct Memory Access) have been widely used due to their low-latency and high-throughput characteristics. RDMA significantly improves network communication efficiency by directly accessing remote memory bypassing the operating system kernel without CPU access. Currently, RDMA-based protocols such as iSER have been widely applied in distributed storage scenarios.

[0022] As Figure 1 shown Figure 1 This is a schematic structural diagram of a distributed storage system based on RDMA dual network cards provided by an embodiment of the present application. Taking a distributed storage cluster composed of two storage nodes as an example, each of node A and node B has two RDMA network cards, which are managed through the Bond technology of the Linux system. Network card 1 is the primary network card, and network card 2 is the standby network card. The iSCSI client and the server TGT service are interconnected through the iSER protocol, and data communication is carried out between storage node A and node B through an RDMA connection. The iSER protocol combines the traditional iSCSI protocol with RDMA and is used for block storage data transmission between the front-end client and the storage system, supporting high-concurrency and low-latency IO operations.

[0023] However, the hardware dependence of RDMA (such as network card queue pairs (Queue Pair, QP), memory registration mechanism) poses severe challenges in the scenario of network card failure. The establishment of an RDMA connection depends on the queue pair and memory registration of the physical network card. Once the primary network card fails, the QP needs to be re-initialized, resulting in connection interruption and service stagnation. At the same time, the session state of iSER is deeply bound to the hardware. After a failure, the context needs to be completely rebuilt, and the recovery process is complex. During this period, communication between nodes is completely interrupted, seriously affecting the IO service quality provided by the distributed storage cluster. The impact of these IO fluctuations is unacceptable to users in scenarios such as finance and real-time analysis.

[0024] Through the present application, it includes obtaining a large page memory pool pre-allocated for each network card; in response to receiving a connection request from a storage client, determining a target polling thread; in response to the target polling thread detecting a network card failure switching event, obtaining the target network card for establishing a connection, and applying for large page memory from the large page memory pool pre-allocated for the target network card. The large page memory is used to initialize remote direct memory access resources, solving the technical problem that in the related solution, when the network card fails to switch, the remote direct memory access resources need to be re-initialized, resulting in connection interruption and service stagnation, and achieving the technical effect of immediately using the large page memory pool of the target network card after the network card fails to switch, without waiting for the resources to be re-created, accelerating the establishment of the connection, and improving service continuity.

[0025] A network card failure switching method provided by an embodiment of the present disclosure, the execution subject of which may be a software module or service process running on a computer system, such as a network driver, a resource manager, etc., and can be applied to a distributed storage system, a high-performance computing cluster, etc.

[0026] In order to enable those skilled in the art of this technology to better understand the solution of the present application, the following further elaborates on the present application in conjunction with the accompanying drawings and specific implementation manners.

[0027] Figure 2A flowchart of a network card failure switching method provided by an embodiment of the present disclosure.

[0028] like Figure 2 As shown, the method comprises the following steps: Step 201, obtaining the large page memory pool pre-allocated by each network card; In some embodiments, the traditional memory page size is usually 4KB, while the large page memory can be 2MB or even 1GB. Using large page memory can reduce the number of translation lookaside buffer (TLB) misses, thereby improving memory access speed.

[0029] In some embodiments, in order to optimize RDMA operations, a large page memory pool is usually pre-allocated to each network card for registering memory regions (Memory Regions, MRs), queues, and other resources.

[0030] In some embodiments, each network card has an independent large page memory pool to avoid performance degradation caused by cross-node access. The memory pool can be pre-allocated at system startup or application initialization stage to reduce dynamic allocation overhead at runtime.

[0031] Step 202, in response to receiving a connection request from a storage client, determining a target polling thread; In some embodiments, a storage client refers to an entity that initiates a data read and write request to a backend storage service. The storage client may be a virtual machine, a container application, a computing node, or a database / file system component.

[0032] In some embodiments, the polling thread is used to monitor network events, such as new connections, data arrival, completion events, and other working threads. The polling thread in this application takes the poller thread as an example.

[0033] In some embodiments, a polling thread with the least number of connections may be determined from multiple polling threads as a target polling thread. During a failover, the target polling thread may detect a change in the network card state and trigger a recovery process.

[0034] Step 203, in response to the target polling thread monitoring the network card failover event, obtain the target network card for establishing the connection, and apply for large page memory from the large page memory pool pre-allocated by the target network card, the large page memory is used to initialize the remote direct memory access resource.

[0035] In some embodiments, a network card failure switching event refers to a process in which a primary network card fails and the system automatically switches to a backup network card, wherein the primary network card failure may be a link disconnection, a driver abnormality, or other failure.

[0036] In some embodiments, the target network card refers to the new active network card after failover. All new connections are bound to this network card and use its corresponding resources, such as large page memory pools, RDMA contexts, etc.

[0037] In some embodiments, as Figure 3 shown, Figure 3 FIG. is a schematic structural diagram of network card failover in a distributed storage system based on RDMA dual network cards provided by an embodiment of the present application. Among them, the main network card of network card 1 of node A fails, and network card 2 is switched to the main network card. The iSER connection between the iSCSI client and the TGT and the RDMA connection between the storage services on nodes A and B need to be switched from network card 1 to network card 2.

[0038] In some embodiments, the distributed storage system provides an external redirect service. When a client wants to connect to the storage system, it first obtains the storage nodes that can be connected through the redirect service. The storage cluster will select one of the nodes as the VIP node to provide the redirect service. When the iSCSI client connects to the storage service, it uses a multi-path method, that is, there are multiple paths between a client and a server, and each path has a connection. The normal process is that the establishment of each path requires obtaining a redirect message from the storage service. After obtaining the redirect information returned by the server, it then connects to the corresponding storage node. Therefore, the establishment of the connection between the client and the server needs to be completed for all paths. When the primary and secondary network cards are switched due to an RDMA network card failure, the iSCSI client recognizes the RDMA link exception and needs to disconnect all the connections of all paths, and then obtain the redirect information from the redirect service again for a new connection. This process takes a long time and affects the continuity of business IO. To address the problems existing in the above process, the present application changes the requirement of obtaining redirect information for the establishment of each path to only the first path obtaining the redirect information from the redirect service, and the remaining paths directly reuse the redirect information of the first path, which can significantly shorten the time required for establishing connections for all links and ensure the continuity of IO issuance during network card failover.

[0039] In some embodiments, remote direct memory access resources may include resources such as queue pairs, memory regions, completion queues, and protection domains. These resources need to be re-initialized after network card switching to support new connections.

[0040] In some embodiments, when the distributed storage system detects a network card failover event, the target polling thread senses the event, determines the target network card, applies for memory from the large page memory pool corresponding to this network card, and initializes the RDMA resources using this memory. Subsequent new connections will use these resources to establish communication, and the old connections can be gradually closed and the resources released back to the pool.

[0041] Through this application, it includes obtaining the large page memory pool pre-allocated for each network card; in response to receiving a connection request from a storage client, determining a target polling thread; in response to the target polling thread detecting a network card failover event, obtaining the target network card for establishing a connection, and applying for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize remote direct memory access resources, solving the technical problem in related solutions that when a network card fails over, the remote direct memory access resources need to be re-initialized, resulting in connection interruption and service stagnation, achieving the technical effect of immediately using the large page memory pool of the target network card after the network card fails over, without waiting for resource re-creation, accelerating the establishment of the connection, and enhancing service continuity.

[0042] In some embodiments, obtaining the large page memory pool pre-allocated for each network card includes: Obtaining the configuration information of the large page memory pool and the network card set associated with the first interface in the distributed storage system; In some embodiments, the configuration information of the large page memory pool can be obtained by parsing the configuration file of the storage service. The configuration information may include the type and size of the large page memory. The configuration information of the large page memory pool can specify a parameter set for how to divide the large page memory, such as how many large pages are allocated to each network card, the large page size (such as 2MB, 1GB), the type of the large page memory, and whether the memory pool can be dynamically expanded, etc.

[0043] In some embodiments, the size of the memory pool can be dynamically adjusted according to the actual business pressure.

[0044] In some embodiments, the distributed storage system refers to a storage architecture where data is distributed across multiple nodes and accessed through a network.

[0045] In some embodiments, the first interface refers to the main network interface of the node for external services in the distributed storage system, usually a logical interface, such as a bond interface. The network card set refers to a list of one or more physical network cards associated with the first interface, which can be used in scenarios such as primary-backup redundancy, load balancing, or multi-path RDMA, as described above Figure 3 where the network card set in the foregoing is network card 1 and network card 2.

[0046] Based on the configuration information of the large page memory pool, determine the large page memory corresponding to each network card in the network card set; In some embodiments, according to the size and type of the large page memory pool in the configuration information of the large page memory pool, apply for large page memory of the corresponding size for each network card entity.

[0047] Perform pooling processing on the large page memory corresponding to each network card to obtain the large page memory pool pre-allocated for each network card.

[0048] In some embodiments, by pooling the huge page memory corresponding to each network card, the huge page memory is divided into blocks of a fixed size, and a free list and allocation / release logic are maintained to achieve efficient memory management and avoid fragmentation.

[0049] In some embodiments, the network card entity information associated with the system bond interface is obtained. According to the size and type of the huge page memory pool parsed in the previous step, huge page memory of the corresponding size is applied for each network card entity, and it is pooled and managed according to the type. In this way, each poller thread is associated with the primary and standby network cards under the bond interface, and each network card is associated with an exclusive huge page memory pool.

[0050] In some embodiments, by obtaining the network card set associated with the first interface and the huge page memory pool configuration information, exclusive huge page memory resources are pre-allocated and pooled and managed for each network card, thereby realizing memory localization, improving RDMA performance and system fault tolerance.

[0051] In some embodiments, obtaining the configuration information of the huge page memory pool includes: In response to the startup of the storage service in the distributed storage system, obtain the configuration file of the storage service; In some embodiments, the storage service refers to the service program running on each storage node, which is responsible for receiving storage client requests and processing data reading and writing; the configuration file of the storage service refers to the configuration file in text or JSON / YAML format loaded when the storage service starts, and the configuration file includes network parameters, the number of threads, the size of the memory pool, and RDMA configuration, etc.

[0052] Parse the configuration file to determine the polling thread set, and the polling thread set includes at least two polling threads; In response to the startup of the polling thread, determine the configuration information of the huge page memory pool from the configuration file of the storage service.

[0053] In some embodiments, when the distributed storage service starts, load the configuration file, parse the configuration content, extract the thread-related configuration items and memory-related configuration items, start multiple Poller threads, and bind them to different network cards or event loops respectively. According to the configuration, an exclusive huge page memory pool is pre-allocated for each network card for the subsequent rapid initialization of RDMA resources (MR, QP).

[0054] In some embodiments, when the storage service of each node of the storage service starts, the TGT module first parses the number of poller threads according to the configuration file. The poller threads are mainly responsible for the resource management of iSCSI client connections and the sending and receiving of data and events. After the poller threads start, by parsing the configuration file, they obtain the type and size of their respective huge page memory pools, and can dynamically adjust the size of the memory pool according to the actual business pressure.

[0055] In some embodiments, by loading a configuration file during the storage service startup phase, dynamically constructing a polling thread set, and initializing a large page memory pool, flexible resource configuration and high-performance network communication support are achieved, which is applicable to a distributed storage system with multiple network cards and RDMA acceleration.

[0056] In some embodiments, after pooling the large page memory corresponding to each network card to obtain the pre-allocated large page memory pool for each network card, the network card failure switching method further includes: Creating corresponding Remote Direct Memory Access (RDMA) resources for the polling threads, where the RDMA resources include at least one of a protection domain and a completion queue.

[0057] In some embodiments, a protection domain (PD) is a resource isolation mechanism in RDMA. All registered memory regions (MRs) and queue pairs (QPs) must belong to a certain PD. The protection domain is used to implement memory access permission control, aiming to prevent illegal access to the memory of other processes or devices.

[0058] In some embodiments, a completion queue (CQ) is a data structure in RDMA for receiving operation completion events. After each work request (WR) is completed, a completion event is generated and placed in the corresponding CQ, and the polling threads periodically check and process these events.

[0059] In some embodiments, after creating corresponding RDMA resources for the polling threads, resources such as PD and CQ are saved in a thread-private structure to reduce cross-thread access overhead and improve concurrency performance.

[0060] In some embodiments, by binding exclusive RDMA resources (such as protection domains, completion queues, etc.) to each polling thread, resource isolation between threads and an efficient event processing mechanism are achieved, which is applicable to multi-threaded RDMA network communication and distributed storage systems.

[0061] In some embodiments, in response to receiving a connection request from a storage client, determining the target polling thread includes: In response to receiving a connection request from a storage client, obtaining the resource occupancy value and business pressure value of each polling thread in the polling thread set; In some embodiments, the resource occupancy value represents the resource usage of the thread pool where the thread is located or the network card / device to which it is bound, including but not limited to CPU usage rate, memory occupancy, network bandwidth, and queue depth, etc. The business pressure value represents the intensity of the business load currently processed by the thread, including but not limited to the current number of connections, the number of unprocessed events, the request queue length, and latency metrics, etc.

[0062] Determine a target polling thread from a set of polling threads based on a resource occupancy value and a service pressure value.

[0063] In some embodiments, the load scores of each thread are comprehensively calculated. The lower the score, the more idle the thread is. The optimal thread is selected as the target thread, and the connection is handed over to this thread for processing. This thread is the target polling thread.

[0064] In some embodiments, based on a resource occupancy value and a service pressure value, a target polling thread is determined from a set of polling threads. The network card failure switching method further includes updating the load status of this thread.

[0065] In some embodiments, by dynamically collecting resource occupancy and service pressure metrics of polling threads when a connection request is received, and accordingly selecting the most suitable target thread, efficient load balancing and resource scheduling are achieved, which is applicable to multi-threaded and high-concurrency distributed storage systems.

[0066] In some embodiments, in response to the target polling thread detecting a network card failure switching event, after obtaining the target network card for establishing a connection and applying for large page memory from the large page memory pool pre-allocated by the target network card, the network card failure switching method further includes: Closing the connection associated with the network card in the standby mode; In some embodiments, the network card in the standby mode refers to a physical or virtual network card that is not currently used for data transmission and is only used as a backup for failure switching. When the primary network card fails, the standby network card will be promoted to the primary network card to continue providing services.

[0067] In some embodiments, a connection represents a communication channel between a client and a server, and is usually represented by a queue pair in RDMA.

[0068] Return the large page memory corresponding to the connection to the large page memory pool pre-allocated by the network card, and release the queue pair resources applied for the connection.

[0069] In some embodiments, when network card failover occurs, the new connection establishment will obtain the network card information where the connection is located, then apply for memory from the associated network card memory pool and initialize relevant resources. The failed network card switches to the Slave mode (standby mode), and the network card associated with the newly incoming connection becomes the Active mode. The poller polling thread actively closes all the connections established under the network card in the Slave mode when it detects the switch of the master and slave network cards. After the connections are closed, all memory resources will be recycled and put back into the free memory pool for subsequent use. At the same time, the QP resources applied for each connection will be destroyed. The RDMA resources such as PD and CQ associated with the poller thread do not need to be recycled. When the network card in the Slave mode switches back to the Active mode later, the RDMA resources such as PD and CQ can be directly reused, which can speed up the connection.

[0070] In some embodiments, when the system detects a network card switch event and the original primary network card becomes the standby state, it traverses all the connections related to the standby network card, closes these connections, clears the RDMA queue pair (QP), terminates the ongoing data transmission, recycles the resources used by the connections, and returns the large page memory to the corresponding large page memory pool. Releases the RDMA resources such as QP and MR, updates the connection status table and resource usage statistics, and reserves resources for subsequent new connections to support seamless switching.

[0071] In some embodiments, as Figure 4 shown, Figure 4 is a schematic flowchart of a method implemented by a storage server in a distributed storage system provided by an embodiment of the present application. Among them, the step of large page memory pooling is the step of initializing the polling thread. After the initialization is completed, it waits for the connection of the storage client. If there is a connection, a corresponding polling thread will be allocated for this connection.

[0072] In some embodiments, by actively closing the connections carried by the network card when it enters the standby mode and recycling the large page memory and RDMA queue pair resources used by it, the efficient reuse of resources and the system fault tolerance are achieved, which is applicable to multi-path RDMA and distributed storage systems.

[0073] In some embodiments, the network card failover method further includes: Initializing the connection information corresponding to each node in the distributed storage system, where the connection information includes at least a status bit; In some embodiments, a node refers to an independent computing unit in the system, usually corresponding to a server or a container instance. Each node may establish multiple connections with other nodes to achieve data transmission and synchronization.

[0074] In some embodiments, in the embodiments of the present application, taking the connection information as an example of the connection structure, it is a data structure for describing the communication status between two nodes, recording the information and status of the forwarding node. The connection information may include, but is not limited to, the current connection status (active / disconnected / migrating), the sending thread status, the receiving thread status, the last heartbeat timestamp, and the fault counter, etc.

[0075] In some embodiments, the status bit is used to indicate the status of the current connection, including at least normal connection and connection unavailable.

[0076] In response to the target polling thread detecting a network card failure switch event, determine the sending thread status and the receiving thread status; Based on the sending thread status and the receiving thread status, determine the status bit in the connection information corresponding to the node.

[0077] In some embodiments, when a network card failure switch occurs, the sending thread fails to send data, and marks the status bit in the corresponding connection structure as dead.

[0078] In some embodiments, when a network card failure switch occurs, the receiving thread detects an abnormality of the corresponding file descriptor fd through the epoll_wait interface, and marks the status bit in the connection information corresponding to the node, such as the status bit in the connection structure, as dead.

[0079] In some embodiments, by initializing the connection information for each node and dynamically evaluating the sending and receiving thread statuses during a network card failure switch, the status bit of the connection is accurately updated, realizing the fine management and high availability guarantee of the communication status in the distributed storage system.

[0080] In some embodiments, after determining the status bit in the connection information corresponding to the node based on the sending thread status and the receiving thread status, the network card failure switch method further includes: In response to receiving an input / output request, obtain the status bit in the connection information corresponding to the node; In some embodiments, the input / output request refers to read and write operations initiated by the client or inside the system. In the distributed storage system, it usually includes data block read requests, data block write requests, and metadata operations, etc.

[0081] If the status bit in the connection information corresponding to the node is unavailable, add the input / output request to the request forwarding failure queue, and add a timestamp identifier to the connection information corresponding to the node.

[0082] In some embodiments, the request forwarding failure queue is used to store I / O requests that cannot be immediately processed due to the unreachability of the target node connection.

[0083] In some embodiments, the timestamp identifier is used to record the earliest occurrence time when the connection is unavailable, and can be used for subsequent timeout judgment or priority sorting.

[0084] In some embodiments, when forwarding the IO requests completed between storage nodes, the corresponding connection structure is obtained first. If the status bit of the connection is marked as dead or the data sending fails, the IO request is put into the forwarding failure queue for caching, and a timestamp is added to the IO request at the same time.

[0085] In some embodiments, by checking the connection status of the target node when receiving an IO request, if the connection is unavailable, it is added to the failure queue and a timestamp is marked in the connection information, realizing the request caching for unreachable nodes and the subsequent retry mechanism, and improving the fault tolerance and stability of the system.

[0086] In some embodiments, if the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to the request forwarding failure queue, and after adding a timestamp identifier to the connection information corresponding to the node, the network card failure switching method further includes: Using the reconnection mechanism to re - establish the connection with the storage client; In response to the successful connection with the storage client, updating the status bit in the connection information corresponding to the node.

[0087] In some embodiments, the reconnection mechanism is a mechanism that automatically attempts to re - establish a communication channel after a network connection is interrupted, and is commonly used in protocol stacks such as TCP and RDMA.

[0088] In some embodiments, when it is detected that the connection is unavailable or a reconnection is actively triggered, the reconnection logic is executed, such as RDMA QP reconstruction, etc. If the connection is successfully established, the status bit in the connection information is updated, and the previous information such as the timestamp and error counter is cleared.

[0089] In some embodiments, by starting the reconnection mechanism after the connection is disconnected and timely updating the connection status bit after the connection is restored, the automatic recovery and status synchronization of the network connection in the distributed storage system are realized, and the fault tolerance and availability of the system are improved.

[0090] In some embodiments, if the status bit in the connection information corresponding to the node is unavailable, the input / output request is added to the request forwarding failure queue, and after adding a timestamp identifier to the connection information corresponding to the node, the network card failure switching method further includes: Obtaining the timestamp corresponding to each request in the request forwarding failure queue; Based on the timestamp corresponding to each request, determining the first request and discarding the first request, where the first request is a timeout request.

[0091] In some embodiments, the timestamp corresponding to each request is used in this scenario to record the time when the request joins the failure queue.

[0092] In some embodiments, the first request refers to a request that has existed in the failure queue for a time exceeding a set threshold. Such requests should be actively discarded to avoid resource occupation.

[0093] In some embodiments, discarding the first request means removing the first request from the failure queue and releasing the resources it occupies. At the same time, the upper-layer application can be notified that the current request has failed.

[0094] In some embodiments, traverse all requests in the request forwarding failure queue, obtain the timestamp information of each request, calculate the difference between the current time and the time when the request enters the queue. If the difference is greater than the set timeout threshold, such as 60 seconds, then discard the corresponding request.

[0095] In some embodiments, by setting timestamps for each request in the request forwarding failure queue and judging whether it times out according to the difference between the current time and the enqueue time, the identification and automatic discarding of timeout requests are realized, thereby effectively controlling memory occupation and enhancing the robustness and resource utilization rate of the system.

[0096] In some embodiments, after obtaining the timestamp corresponding to each request in the request forwarding failure queue, the network card failure switching method further includes: Based on the timestamp corresponding to each request, determine a second request, where the second request is a non-timeout request; Determine the status bit in the connection information corresponding to the second request; In response to the status bit in the connection information corresponding to the second request being available, forward the second request and delete the second request from the request forwarding failure queue.

[0097] In some embodiments, the second request refers to an I / O request in the request forwarding failure queue that has not exceeded the set timeout time, indicating that the request may be successfully forwarded because the target node has recovered.

[0098] In some embodiments, traverse all requests in the failure queue, judge whether it times out according to the timestamp. If it does not time out, then mark it as the "second request", obtain the connection information of the target node corresponding to the request, check whether the connection status bit is in the "available" state. If the connection is available, then try to forward the request. If the forwarding is successful, then delete the request from the failure queue and release its resources.

[0099] In some embodiments, through timestamp screening and connection status judgment of requests in the failure queue, automatically retry non-timeout requests after the connection is restored and delete them from the failure queue, realizing the intelligent recovery and resource recycling of abnormal requests in the distributed storage system.

[0100] Through the present application, it includes obtaining the large page memory pool pre-allocated for each network card; in response to receiving a connection request from a storage client, determining a target polling thread; in response to the target polling thread detecting a network card failover event, obtaining the target network card for establishing a connection, and applying for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize remote direct memory access resources, solving the technical problem that in the related solution, when a network card fails over, the remote direct memory access resources need to be re-initialized, resulting in connection interruption and service stagnation, achieving the technical effect of immediately using the large page memory pool of the target network card after the network card fails over, without waiting for the resources to be re-created, accelerating the establishment of the connection, and enhancing service continuity.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0102] An embodiment of the present application further provides a network card failover device 500. Figure 5 It is a schematic structural diagram of a network card failover device provided by an embodiment of the present disclosure, as Figure 5 shown, including: An obtaining unit 501, which obtains the large page memory pool pre-allocated for each network card; A determining unit 502, configured to determine a target polling thread in response to receiving a connection request from a storage client; An applying unit 503, configured to obtain the target network card for establishing a connection in response to the target polling thread detecting a network card failover event, and apply for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize remote direct memory access resources.

[0103] Further, in a possible implementation manner of an embodiment of the present disclosure, the obtaining unit 501 is used for: Obtaining the configuration information of the large page memory pool and the network card set associated with the first interface in the distributed storage system; Based on the configuration information of the large page memory pool, determining the large page memory corresponding to each network card in the network card set; Performing pooling processing on the large page memory corresponding to each network card to obtain the large page memory pool pre-allocated for each network card.

[0104] Further, in a possible implementation manner of an embodiment of the present disclosure, the obtaining unit 501 is used for: In response to the start of the storage service in the distributed storage system, obtaining the configuration file of the storage service; Parse the configuration file to determine a polling thread set, where the polling thread set includes at least two polling threads; In response to the startup of the polling thread, determine the configuration information of the large page memory pool from the configuration file of the storage service.

[0105] Further, in a possible implementation manner of the embodiment of the present disclosure, the network card failover device 500 further includes a resource creation unit, and the resource creation unit is used for: Create corresponding remote direct memory access resources for the polling threads, where the remote direct memory access resources include at least one of a protection domain and a completion queue.

[0106] Further, in a possible implementation manner of the embodiment of the present disclosure, the determining unit 502 is used for: In response to receiving a connection request from the storage client, obtain the resource occupancy value and the service pressure value of each polling thread in the polling thread set; Based on the resource occupancy value and the service pressure value, determine a target polling thread from the polling thread set.

[0107] Further, in a possible implementation manner of the embodiment of the present disclosure, the network card failover device 500 further includes a resource release unit, and the resource release unit is used for: Close the connection associated with the network card in the standby mode; Return the large page memory corresponding to the connection to the large page memory pool pre-allocated by the network card, and release the queue pair resources applied for the connection.

[0108] Further, in a possible implementation manner of the embodiment of the present disclosure, the network card failover device 500 further includes a status bit determination unit, and the status bit determination unit is used for: Initialize the connection information corresponding to each node in the distributed storage system, where the connection information includes at least a status bit; In response to the target polling thread detecting a network card failover event, determine the sending thread status and the receiving thread status; Based on the sending thread status and the receiving thread status, determine the status bit in the connection information corresponding to the node.

[0109] Further, in a possible implementation manner of the embodiment of the present disclosure, the network card failover device 500 further includes an adding unit, and the adding unit is used for: In response to receiving an input / output request, obtain the status bit in the connection information corresponding to the node; If the status bit in the connection information corresponding to the node is unavailable, add the input / output request to the request forwarding failure queue, and add a timestamp identifier to the connection information corresponding to the node.

[0110] Further, in a possible implementation manner of the embodiments of the present disclosure, the network card failure switching device 500 further includes an updating unit, and the updating unit is configured to: Re - establish a connection with the storage client by using the re - connection mechanism; In response to successfully connecting to the storage client, update the status bit in the connection information corresponding to the node.

[0111] Further, in a possible implementation manner of the embodiments of the present disclosure, the determining unit 502 is configured to: Obtain the time stamp corresponding to each request in the request forwarding failure queue; Based on the time stamp corresponding to each request, determine the first request and discard the first request, where the first request is a timeout request.

[0112] Further, in a possible implementation manner of the embodiments of the present disclosure, the determining unit 502 is configured to: Based on the time stamp corresponding to each request, determine the second request, where the second request is a non - timeout request; Determine the status bit in the connection information corresponding to the second request; In response to the status bit in the connection information corresponding to the second request being available, forward the second request and delete the second request from the request forwarding failure queue.

[0113] Through the present application, including obtaining the large - page memory pool pre - allocated for each network card; in response to receiving a connection request from the storage client, determining the target polling thread; in response to the target polling thread detecting a network card failure switching event, obtaining the target network card for establishing a connection and applying for large - page memory from the large - page memory pool pre - allocated for the target network card, where the large - page memory is used to initialize remote direct memory access resources, the technical problem that in the related solutions, when the network card fails to switch, the remote direct memory access resources need to be re - initialized, resulting in connection interruption and service stagnation is solved, and the technical effect of immediately using the large - page memory pool of the target network card after the network card fails to switch, without waiting for the resources to be re - created, accelerating the establishment of the connection, and improving service continuity is achieved.

[0114] For the description of the features in the embodiments corresponding to the network card failure switching device, reference can be made to the relevant descriptions in the embodiments corresponding to the network card failure switching method, which will not be elaborated here one by one.

[0115] The embodiments of the present application further provide an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above - mentioned embodiments of the network card failure switching method.

[0116] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any of the above-described embodiments of the network card failure switching method when running.

[0117] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0118] An embodiment of the present application further provides a computer program product, where the computer program product includes a computer program, and the computer program implements the steps in any of the above-described embodiments of the network card failure switching method when executed by a processor.

[0119] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores a computer program, and the computer program implements the steps in any of the above-described embodiments of the network card failure switching method when executed by a processor.

[0120] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0121] The above has introduced in detail a network card failure switching method, device, electronic device, storage medium, and product provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A network card failure switching method, characterized in that, Including: Obtain the large page memory pool pre-allocated for each network card; In response to receiving a connection request from a storage client, determine a target polling thread; In response to the target polling thread detecting a network card failover event, obtain the target network card for establishing a connection, and apply for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize remote direct memory access resources.

2. The network card failure switching method according to claim 1, wherein The obtaining the large page memory pool pre-allocated for each network card includes: Obtain the configuration information of the large page memory pool and the set of network cards associated with the first interface in the distributed storage system; Based on the configuration information of the large page memory pool, determine the large page memory corresponding to each network card in the set of network cards; Perform pooling processing on the large page memory corresponding to each network card to obtain the large page memory pool pre-allocated for each network card.

3. The network card failure switching method according to claim 2, wherein, The obtaining the configuration information of the large page memory pool includes: In response to the start of the storage service in the distributed storage system, obtain the configuration file of the storage service; Parse the configuration file to determine a set of polling threads, where the set of polling threads includes at least two polling threads; In response to the start of the polling thread, determine the configuration information of the large page memory pool from the configuration file of the storage service.

4. The network card failure switching method according to claim 3, wherein After performing pooling processing on the large page memory corresponding to each network card to obtain the large page memory pool pre-allocated for each network card, the method further includes: Create corresponding remote direct memory access resources for the polling thread, where the remote direct memory access resources include at least one of a protection domain and a completion queue.

5. The network card failure switching method according to claim 3, wherein The determining a target polling thread in response to receiving a connection request from a storage client includes: In response to receiving a connection request from a storage client, obtain the resource occupancy value and the business pressure value of each polling thread in the set of polling threads; Based on the resource occupancy value and the business pressure value, determine a target polling thread from the set of polling threads.

6. The network card failure switching method according to claim 5, wherein, After obtaining the target network card for establishing a connection and applying for large page memory from the large page memory pool pre-allocated for the target network card in response to the target polling thread detecting a network card failover event, the method further includes: Close the connection associated with the network card in the standby mode; Return the large page memory corresponding to the connection to the large page memory pool pre-allocated for the network card, and release the queue pair resources applied for the connection.

7. The network card failure switching method according to claim 1, wherein The method further includes: Initialize the connection information corresponding to each node in the distributed storage system, where the connection information includes at least a status bit; In response to the target polling thread detecting a network card failover event, determine the send thread status and the receive thread status; Based on the send thread status and the receive thread status, determine the status bit in the connection information corresponding to the node.

8. The network card failure switching method according to claim 7, wherein After determining the status bit in the connection information corresponding to the node based on the send thread status and the receive thread status, the method further includes: In response to receiving an input / output request, obtain the status bit in the connection information corresponding to the node; If the status bit in the connection information corresponding to the node is unavailable, add the input / output request to the request forwarding failure queue, and add a timestamp identifier to the connection information corresponding to the node.

9. The network card failure switching method according to claim 8, wherein After adding the input / output request to the request forwarding failure queue and adding a timestamp identifier to the connection information corresponding to the node if the status bit in the connection information corresponding to the node is unavailable, the method further includes: Use a reconnection mechanism to re-establish a connection with the storage client; In response to a successful connection with the storage client, update the status bit in the connection information corresponding to the node.

10. The network card failure switching method according to claim 8, wherein, After adding the input / output request to the request forwarding failure queue and adding a timestamp identifier to the connection information corresponding to the node if the status bit in the connection information corresponding to the node is unavailable, the method further includes: Obtain the timestamp corresponding to each request in the request forwarding failure queue; Based on the timestamp corresponding to each request, determine a first request and discard the first request, where the first request is a timeout request.

11. The network card failure switching method according to claim 10, characterized in that, After obtaining the timestamp corresponding to each request in the request forwarding failure queue, the method further includes: Based on the timestamp corresponding to each request, determine a second request, where the second request is a non-timeout request; Determine the status bit in the connection information corresponding to the second request; In response to the status bit in the connection information corresponding to the second request being available, forward the second request and delete the second request from the request forwarding failure queue.

12. A network card failure switching device, characterized in that, Includes: An acquisition unit for acquiring the large page memory pool pre-allocated for each network card; A determination unit for determining a target polling thread in response to receiving a connection request from a storage client; An application unit for, in response to the target polling thread detecting a network card failure switch event, acquiring a target network card for establishing a connection, and applying for large page memory from the large page memory pool pre-allocated for the target network card, where the large page memory is used to initialize remote direct memory access resources.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the network card failure switch method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program, when executed by a processor, implements the steps of the network card failure switch method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the network card failure switch method according to any one of claims 1 to 11.

Images