A flow communication and dynamic switching method and device based on three-stack fusion

Abstract

The present invention relates to a flow communication and dynamic switching method and device based on three-stack fusion, belonging to the field of high-performance networks. The method supports the integration of three stacks of RDMA, user-mode TCP and kernel-mode TCP protocols by integrating the protocol stack, can shield the internal implementation differences of different transmission protocols, and provide a unified communication interface for applications. Compared with the protocol stack of a single transmission protocol, the application does not need to pay attention to the internal implementation differences and programming interfaces of each transmission protocol. The method is simple and easy to use, can not only give play to the technical advantages of RDMA and TCP protocols at the same time, but also can solve the problem that large-scale networking cannot be achieved by using only the RDMA protocol.

Description

Technical Field

[0001] The present invention belongs to the field of high-performance networks, and in particular relates to a flow communication and dynamic switching method and device based on three-stack fusion. Background Art

[0002] For many years, TCP / IP protocol has been the mainstay of Internet communications. The traditional kernel TCP protocol stack has stable performance and strong reliability. However, due to the existence of interrupt processing, kernel copy and other mechanisms, the kernel protocol stack has many performance bottlenecks. The user-mode TCP protocol stack can bypass the kernel and directly provide the data of network hardware devices to user-mode applications, which can reduce interrupts and memory copies. Compared with the kernel TCP protocol stack, the performance is greatly improved. However, applications such as high-performance computing and distributed storage require ultra-high bandwidth, ultra-low latency and ultra-high reliability for the network. User-mode TCP still cannot meet the demands of high-performance computing services. RDMA can directly access memory data through the network interface without the intervention of the operating system kernel. Therefore, the industry generally uses RDMA to replace the TCP protocol.

[0003] However, RDMA networks are very sensitive to packet loss. In a lossless state, they can transmit at full rate, but once packet loss and retransmission occur, performance will drop sharply. When nodes within an AZ communicate, congestion and packet loss can be avoided by enabling PFC and the congestion control mechanism of the RDMA network card on the switch to ensure high performance of the RDMA network. However, studies have shown that large-scale RDMA networks are prone to deadlocks and PFC storms, which reduce performance. Therefore, when communicating across AZs, the RDMA protocol is still used, but the effect is not ideal, and RDMA networks are difficult to deploy on a large scale. Summary of the invention

[0004] In view of the problems of the prior art, the present invention provides a traffic communication and dynamic switching method and device based on three-stack fusion, and the technical solution adopted is:

[0005] In a first aspect, a traffic communication and dynamic switching method based on three-stack fusion is provided, the method comprising:

[0006] S01, initializing the fusion protocol stack of each computing node and storage node;

[0007] S02, according to the computing node, recording the time point t1 by sending a negotiation message, and recording the time point t2 according to the reply of the negotiation message;

[0008] S03, judging whether the computing node and the storage node are in the same AZ by the round-trip delay between the time t1 and t2 according to the configured round-trip delay threshold;

[0009] S04, when the computing node and the storage node belong to the same AZ, establishing a read and write request by integrating RDMA as a transmission protocol;

[0010] S14: When the computing node and the storage node do not belong to the same AZ, a read / write request is established using user-mode TCP as a transmission protocol;

[0011] S05, when an abnormality occurs in the transmission protocol of the computing node and the storage node, the computing node migrates the service data to the kernel TCP;

[0012] S06, re-establishing the read and write request between the computing node and the storage node according to S04; and migrating the service data in the kernel TCP to the storage node according to the corresponding transmission type;

[0013] Among them, S06 specifically includes:

[0014] S061, reclaiming QP resources through the computing node and notifying the storage node to reclaim QP resources, and periodically checking the IB network card status;

[0015] S062, when the network card status is restored, re-establishing the RDMA QP channel between the computing node and the storage node;

[0016] S063: According to the RDMA QP channel, the service data is migrated back to the RDMA QP through the computing node.

[0017] Furthermore, S04 specifically includes:

[0018] S041, when the computing node and the storage node are in the same AZ and support integrated RDMA, select integrated RDMA as the transmission protocol of the computing node;

[0019] S042, creating a QP according to the computing node, creating a QP in the storage node through a QP negotiation message, and completing the RDMA channel between the computing node and the storage node;

[0020] S043: According to the RDMA QP channel, the computing node sends a read / write request to the storage node to implement service communication.

[0021] Furthermore, S05 specifically includes:

[0022] S051, the fusion protocol stack monitors that the IB device is abnormal and the QP is in an error state, establishes a kernel-state TCP socket through the computing node, and sends a channel connection request to the storage node to complete the channel connection creation;

[0023] S052, the computing node migrates the business data to the kernel TCP channel and maintains the read-write state.

[0024] Furthermore, S14 specifically includes:

[0025] S141, when the computing node and the storage node are in different AZs, and their RTBB is greater than the round-trip delay threshold and both support user-mode TCP, user-mode TCP is selected as the transmission protocol of the computing node;

[0026] S142, creating a user-mode TCP socket according to the computing node, and initiating a link establishment request to the storage node, and completing the link establishment through the storage node replying to the link establishment request;

[0027] S143, according to the link establishment, a read / write request is sent to the storage node through the computing node to realize business communication.

[0028] Furthermore, S05 specifically includes:

[0029] S151, when a user-mode TCP channel exception occurs in the computing node, the underlying protocol stack closes the user-mode TCP socket and creates a kernel-mode TCP socket, and sends a kernel-mode TCP channel connection request to the storage node to complete the channel creation;

[0030] S152, the computing node migrates the business data to the kernel TCP and maintains the read-write state.

[0031] Further, S06 is replaced by the following steps:

[0032] S161, the computing node recreates a user-mode TCP socket and establishes a user-mode TCP channel with the storage node;

[0033] S162: According to the user-mode TCP channel, the service data is migrated back to the user-mode TCP through the computing node.

[0034] In a second aspect, an embodiment of the present invention provides a traffic communication and dynamic switching device based on three-stack fusion, which is used to implement the traffic communication and dynamic switching method based on three-stack fusion described in the first aspect, and the device includes:

[0035] An external interface module, used to provide a unified calling interface for application software; wherein the application software includes the traffic communication and dynamic switching method based on three-stack fusion;

[0036] A connection management module, used to implement the negotiation of control messages and the adaptive selection of transmission protocols between computing nodes and storage nodes;

[0037] A task scheduling module, used to manage the connection switching of the connection management module and the dynamic loading and unloading of tasks;

[0038] The task execution module is used to encapsulate the data processing in the protocol into tasks and add them to the task execution queue;

[0039] Transmission protocol module; used for sending and receiving processing of each underlying protocol stack, including RDMA protocol processing unit, user-mode TCP protocol processing unit and kernel-mode TCP protocol processing module unit.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the method described in the first aspect above is implemented.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the first aspect.

[0042] The beneficial effects of the present invention are as follows:

[0043] The method of the present invention integrates the protocol stack to shield the internal implementation differences of the three transmission protocols, and provides a unified calling interface for the upper layer services, so as to be simple and easy to use;

[0044] By actively detecting the network environment and adaptively choosing to use RDMA or user-mode TCP based on the round-trip delay between communication nodes, it can fully utilize network resources and improve communication performance without application intervention.

[0045] Monitor the connection status and network card status in real time. Once network congestion or misoperation occurs, causing the original channel to be unavailable, the transmission protocol is immediately switched to the highly reliable kernel-mode TCP channel. After the fault is recovered, the RDMA or user-mode TCP channel is actively restored to migrate the business data back. The business does not need to be aware of the whole process, and business communication can be guaranteed to be uninterrupted, which can greatly improve the stability and reliability of communication.

[0046] By integrating the protocol stack, we can fully leverage the advantages of RDMA's high performance and low latency, and use the TCP protocol as a supplement to solve the problem that RDMA networks are difficult to deploy on a large scale, achieving the goals of simplicity, ease of use, high performance, and high reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0048] Figure 1 A flow chart of adaptive selection of a transmission protocol for a traffic communication and dynamic switching method based on three-stack fusion provided by an embodiment of the present invention;

[0049] Figure 2 It is a flow chart of a transmission protocol fault switching method of a traffic communication and dynamic switching method based on three-stack fusion provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a traffic communication and dynamic switching device based on three-stack fusion provided by an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of the transmission protocol adaptation principle provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0053] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0054] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0056] First, some of the terms involved in the present invention are explained:

[0057] TCP (Transmission Control Protocol): A protocol that provides reliable data transmission in an IP environment.

[0058] RDMA (Remote Direct Memory Access): A technology that can directly read and write data in the memory of other servers without going through the operating system / CPU.

[0059] AZ (Availability Zone) consists of one or several nearby data centers, and AZs are interconnected by a low-latency fiber optic network.

[0060] PFC (Priority-based Flow Control) is a priority-based flow control technology. Queues with PFC enabled are called lossless queues. When congestion occurs in the lossless queue of a downstream device, the downstream device notifies the upstream device to stop sending traffic in the queue, thereby achieving zero packet loss transmission.

[0061] RTT (Round-Trip Time) is the total delay from the time the sender sends data to the time the sender receives confirmation from the receiver.

[0062] QP (Queue Pair, work queue), an abstract description of a connection in RDMA.

[0063] ACK (Acknowledge character) is a transmission control character sent by the receiving side to the sending side in data communication. It indicates that the received data has been confirmed to be received correctly.

[0064] Embodiment 1:

[0065] Figure 1 A flow chart of adaptive selection of transmission protocols for traffic communication and dynamic switching method based on three-stack fusion is shown;

[0066] Figure 2 A flow chart of transmission protocol failure switching based on traffic communication and dynamic switching method of three-stack fusion is shown;

[0067] The traffic communication and dynamic switching method based on three-stack fusion provided in this embodiment includes:

[0068] First, according to S01, the fusion protocol stack of each computing node A and storage node C is initialized;

[0069] Next, according to S02, according to computing node A, the time point t1 is recorded by sending a negotiation message, and the time point t2 is recorded according to the reply of the negotiation message;

[0070] Computing node A creates a QP and sends a QP negotiation message, including the starting sequence number of the QP, the QP number, the local gid and other information. After receiving the negotiation message, storage node C creates a QP, connects to the QP, and replies to the QP negotiation message. After receiving the reply message, computing node A connects to the QP and notifies the service connection that it is available.

[0071] Next, according to S03, based on the configured round-trip delay threshold, it is determined whether the computing node A and the storage node C are in the same AZ by the round-trip delay between the time t1 and t2;

[0072] The round-trip delay RTTA between computing node A and storage node C is calculated as t2-t1. Generally speaking, within the same AZ, the node round-trip delay will be less than 100us, and the cross-AZ node round-trip delay will generally be around 1ms. Therefore, the threshold is configured to 200us. When the round-trip delay is less than the threshold, it can be considered that the communication nodes are in the same AZ, otherwise it is cross-AZ communication.

[0073] Next, according to S041, when the computing node A and the storage node C are in the same AZ and support integrated RDMA, integrated RDMA is selected as the transmission protocol of the computing node A;

[0074] Since RTTA is less than the threshold and both nodes support RDMA, RDMA is used as the transmission protocol.

[0075] Next, according to S042, a QP is created according to computing node A, and a QP is created in storage node C through a QP negotiation message, thereby completing a QP connection between the computing node A and the storage node C;

[0076] Compute node A creates a QP and sends a QP negotiation message, including the starting sequence number of the QP, the QP number, the local gid and other information. After receiving the negotiation message, storage node C creates a QP, connects to the QP, and replies to the QP negotiation message.

[0077] Next, according to S043, based on the QP connection, the computing node A sends a read / write request to the storage node C to implement service communication;

[0078] Computing node A creates an RDMA channel QP and sends a QP negotiation message, including the starting sequence number of the QP, the QP number, the local gid and other information. After receiving the negotiation message, storage node C creates a QP, connects to the QP, and replies to the QP negotiation message. After receiving the reply message, computing node A connects to the QP and notifies the service connection that it is available.

[0079] Next, according to S051, the fusion protocol stack monitors that the IB device is abnormal, the QP is in the error state, establishes a kernel-mode TCP connection through the computing node, and sends a connection request to the storage node to complete the connection creation;

[0080] The data sending task of computing node A cannot select an available data channel, so it notifies the connection management module to immediately create a kernel-state TCP channel and sends a kernel-state TCP channel creation request to storage node C. Storage node C replies to the request, and the kernel TCP channel is successfully created.

[0081] Next, according to S052, the computing node migrates the service data to the kernel TCP and maintains the read-write state;

[0082] That is, when computing node A retransmits service data, it selects the kernel-state TCP channel and migrates the service data to the kernel-state TCP channel to ensure that IO is not interrupted;

[0083] Next, according to S061, the computing node A reclaims the QP resources and notifies the storage node C to reclaim the QP resources, and periodically checks the IB network card status;

[0084] Reclaim QP resources through computing node A, notify storage node C to reclaim QP resources, and periodically check the IB network card status;

[0085] Compute node A resets the QP state and notifies the peer storage node C to perform the reset operation, initializes QP, and attempts to renegotiate QP with storage node C until the QPs at both ends are successfully connected and the RDMA channel state is set to available;

[0086] Next, according to S062, when the network card status is restored, the QP connection between the computing node A and the storage node C is re-established;

[0087] When the network card is restored, try to create a QP until the QP can be created successfully, and try to negotiate the QP with storage node C until the QPs at both ends are successfully connected;

[0088] Finally, according to S063, the service data is migrated back to the RDMAQP through the computing node A based on the QP connection.

[0089] That is, the business data is migrated back to the RDMA QP through computing node A.

[0090] The data sending task of computing node A is selected to the RDMA channel, and the business data is migrated back to the RDMA channel;

[0091] Computing node A detects that the kernel-state TCP channel has not been used for a long time, destroys the kernel-state TCP channel, and sends a destruction request to storage node C. Storage node C simultaneously destroys the kernel-state TCP channel.

[0092] Embodiment 2:

[0093] Figure 1 A flow chart of adaptive selection of transmission protocols for traffic communication and dynamic switching method based on three-stack fusion is shown;

[0094] Figure 2 A flow chart of transmission protocol failure switching based on traffic communication and dynamic switching method of three-stack fusion is shown;

[0095] The traffic communication and dynamic switching method based on three-stack fusion provided in this embodiment includes:

[0096] First, according to S01, the fusion protocol stack of each computing node B and storage node C is initialized;

[0097] Next, according to S02, according to computing node B, the time point t1 is recorded by sending a negotiation message, and the time point t2 is recorded according to the reply of the negotiation message;

[0098] Computing node B creates a QP and sends a QP negotiation message, including the starting sequence number of the QP, the QP number, the local gid and other information. After receiving the negotiation message, storage node C creates a QP, connects to the QP, and replies to the QP negotiation message. After receiving the reply message, computing node B connects to the QP and notifies the service connection that it is available.

[0099] Next, according to S03, based on the configured round-trip delay threshold, it is determined whether the computing node B and the storage node C are in the same AZ by the round-trip delay between the time t1 and t2;

[0100] The round-trip delay RTTA between computing node B and storage node C is calculated as t2-t1. Generally speaking, within the same AZ, the node round-trip delay will be less than 100us, and the cross-AZ node round-trip delay will generally be around 1ms. Therefore, the threshold is configured to be 200us. When the round-trip delay is less than the threshold, it can be considered that the communication nodes are in the same AZ, otherwise it is cross-AZ communication.

[0101] Next, according to S141, when the computing node B and the storage node C are in different AZs, and their RTBB is greater than the round-trip delay threshold and both support user-mode TCP, user-mode TCP is selected as the transmission protocol of the computing node B;

[0102] By obtaining RTTB, since RTTB is greater than the threshold threshold, it indicates that computing node B and storage node C cross AZs, and both computing node B and storage node C support user-mode TCP, so computing node B uses user-mode TCP as the transmission protocol.

[0103] Next, according to S142, a user-mode TCP socket is created according to computing node B, and a link establishment request is initiated to storage node C, and the link establishment is completed by the storage node C replying to the link establishment request;

[0104] Next, according to S143, based on the link establishment, a read / write request is sent to the storage node C through the computing node B to realize business communication.

[0105] Computing node B uses the connection created in S142 to call the data sending interface and send a read / write IO request to storage node C. The data sending task selects the user-state TCP channel and delivers the data from the user-state TCP channel to the network. Storage node C receives the data from the user-state TCP channel, and the service achieves normal communication.

[0106] Next, according to S151, when the computing node B has a user-mode TCP channel exception, the underlying protocol stack closes the user-mode TCP socket and creates a kernel-mode TCP socket, and sends a kernel-mode TCP connection request to the storage node C to complete the TCP channel creation;

[0107] Compute node B creates a user-mode TCP socket and initiates a link establishment request to storage node C. Storage node C replies to the link establishment request. The link establishment is successful, and compute node B notifies the service that the connection is available.

[0108] Next, according to S152, the computing node B migrates the service data to the kernel TCP and maintains the read-write state.

[0109] Computing node B calls the message sending interface through the connection created in step S151 to send a read / write IO request to storage node C, and the business achieves normal communication.

[0110] Next, according to S061, the computing node B creates a user-mode TCP socket and establishes a user-mode TCP connection with the storage node C;

[0111] Compute node B finds that the user-mode TCP channel is abnormal, the underlying protocol stack closes the TCP socket, immediately creates a kernel-mode TCP socket, and sends a kernel-mode TCP connection request to storage node C. Storage node C replies to the request, and the channel is successfully created.

[0112] Computing node B detects an abnormality in the user-mode TCP socket and sets the user-mode TCP channel to an unavailable state.

[0113] The computing node B data sending task cannot select an available data channel, so it notifies the connection management module to immediately create a kernel-state TCP channel and sends a kernel-state TCP channel creation request to the storage node C. The storage node C replies to the request, and the kernel-state TCP channel is successfully created and set to the available state.

[0114] When computing node B retransmits service data, it selects the kernel-state TCP channel and immediately migrates the service data to the kernel-state TCP channel to ensure that IO is not interrupted.

[0115] Next, according to S062, the service data is migrated back to the user state TCP through the computing node B according to the user state TCP channel;

[0116] Compute node B attempts to recreate a user-mode TCP socket. After the creation is successful, it attempts to establish a user-mode TCP connection with storage node C until the connection is successfully established. Then, the business data is migrated back to the user-mode TCP connection through compute node B.

[0117] Compute node B destroys the faulty user-state TCP socket, recreates the user-state TCP socket, and attempts to re-establish a user-state TCP connection with storage node C until both ends are successfully connected, setting the user-state TCP channel to an available state.

[0118] The data sending task of computing node B is selected to the user-mode TCP channel, and the business data is migrated back to the user-mode TCP channel;

[0119] Compute node B detects that the kernel-state TCP channel has not been used for a long time, destroys the kernel-state TCP channel, and sends a destruction request to storage node C. Storage node C simultaneously destroys the kernel-state TCP channel

[0120] The present invention detects the RTT between communication nodes, determines whether the communicating nodes are located in the same AZ according to the RTT, and adaptively selects a suitable transmission protocol for the communication nodes without application intervention, which can fully utilize network resources and improve communication performance.

[0121] At the same time, the connection status and network card status are monitored in real time. Once network congestion or network card abnormality occurs, causing the original connection to be unavailable, the transmission protocol is immediately switched to the highly reliable kernel-mode TCP protocol. After the fault is recovered, it actively restores to the RDMA or user-mode TCP connection to re-migrate the business data. The business does not need to be aware of the whole process, and business communication can be guaranteed to be uninterrupted, which can greatly improve the communication stability and reliability;

[0122] The present invention supports the integration of RDMA, user-mode TCP and kernel-mode TCP protocols by integrating the protocol stack, which can shield the internal implementation differences of different transmission protocols and provide a unified communication interface for applications. Compared with the protocol stack of a single transmission protocol, the application does not need to pay attention to the internal implementation differences and programming interfaces of each transmission protocol. It is simple and easy to use, and can not only give full play to the technical advantages of RDMA and TCP protocols at the same time, but also solve the problem that large-scale networking cannot be achieved by using only the RDMA protocol.

[0123] Embodiment three:

[0124] Figure 2A schematic diagram of a flow communication and dynamic switching device based on three-stack fusion is shown, in order to implement the method described in Example 1, as shown in FIG. Figure 2 As shown, the flow communication and dynamic switching device based on three-stack fusion provided in this embodiment includes:

[0125] External interface module: used to provide a unified calling interface for application software; wherein, the application software includes the traffic communication and dynamic switching method based on three-stack fusion; the external interface module includes the fusion protocol stack resource initialization interface, connection creation interface, data transmission interface, and task processing interface. The resource initialization interface initializes the resources of each underlying protocol stack; the connection creation interface creates a connection from the client node to the server node, and returns the connection number to the application software; the application software calls the data transmission interface to send data from a specified connection number; the application software cyclically calls the task processing interface to process various events in the fusion protocol stack, such as asynchronous data transmission, data reception, etc.

[0126] The connection management module is used to implement control message negotiation between the two communication nodes, adaptive selection of transmission protocols, data connection creation, status monitoring and fault switching, etc.

[0127] The fusion transport stack gives priority to RDMA for communication nodes within an AZ and user-mode TCP for cross-AZ communication nodes. This is mainly because the transmission delay of the communication link channel across AZs is large and it is difficult to ensure lossless transmission. Considering that RDMA is very sensitive to packet loss, RDMA is no longer applicable to communication between nodes across AZs.

[0128] Adaptive selection of transmission protocol: Supports active detection of RTT between communication nodes, and determines whether the communication nodes are located in the same AZ based on RTT. Generally speaking, the round-trip delay of nodes in the same AZ will be less than 100us, and the round-trip delay of nodes across AZs will generally be around 1ms. Therefore, the threshold is configured to 200us. When RTT is less than the threshold, it can be considered that the communication nodes are in the same AZ, otherwise it is cross-AZ communication (200us is not an absolute situation and can be used as a parameter to be configured according to the actual network environment).

[0129] Data connection creation: A connection in the fusion protocol stack is an abstract description of the communication connection from the client node to the server node, and is managed by a connection number. Multiple data channels are managed under one connection. A channel is a communication instance of a specific transmission protocol, such as a QP under the RDMA transmission protocol, or a TCP socket under the TCP transmission protocol. When the connection is initialized, only high-priority channels, such as RDMA channels or user-mode TCP channels, will be created. Only when a high-priority channel becomes unavailable due to a fault will a kernel-mode TCP channel be created. When the high-priority channel becomes available again, the kernel-mode TCP channel will be destroyed in time to save resources.

[0130] Connection status monitoring: Real-time monitoring of the connection status on the node, setting the channel management state machine, and setting the channel status to unavailable when a failure occurs. In this way, when data needs to be sent, no available RDMA channel or user-mode TCP channel can be selected, thus triggering the creation of a kernel-mode TCP channel to ensure that business traffic is not interrupted.

[0131] Connection automatic repair: The fusion protocol stack has a connection automatic repair function. When the channel status is unavailable, it will periodically try to restore the error channel by resetting the error channel or creating a new channel. When the high-priority channel is restored, the channel status is set to available. When there is subsequent data to be sent, the data can be moved back to the RDMA or user-mode TCP channel to achieve efficient and stable business transmission.

[0132] A task scheduling module, used to manage the connection switching of the connection management module and the dynamic loading and unloading of tasks;

[0133] Dynamic loading and unloading of tasks are implemented based on the channel usage in the connection management module. For example, when only RDMA-type channels are created on this node, the task scheduling module will only load RDMA-type tasks for processing. When a kernel-state TCP channel is created when a fault occurs, kernel-state TCP-type tasks will be loaded for processing. When the fault is recovered and the kernel-state TCP channel is destroyed, kernel-state TCP-type tasks will also be unloaded, so as to avoid the waste of CPU resources caused by the CPU executing invalid tasks as much as possible.

[0134] The task execution module is a high-performance processing module on the data plane. The data sending and receiving processing in each protocol are abstractly encapsulated into tasks, which are added to the task execution queue through the task calling module. The application software cyclically calls the task processing interface to trigger task execution. There are four main types of tasks:

[0135] Asynchronous data transmission task: In order to ensure reliable data transmission, the fusion protocol stack creates an asynchronous transmission queue with the working thread as the granularity. When the application software calls the data transmission interface, it only queues the data into the asynchronous transmission queue. The data transmission task will try to deliver the data in the asynchronous transmission queue to the network in turn, and give priority to selecting high-priority channels from the specified connection for transmission. Only when the high-priority channel is not available, it will check whether the kernel-state TCP channel is available. If the kernel-state TCP channel is available, it will be sent directly from the kernel-state TCP channel; if the kernel-state TCP channel is not created at this time, it will notify the connection management module to create it; when all channels are unavailable, it will periodically try to send until the data packet is delivered to the network.

[0136] Data sending successful confirmation task: The underlying transmission protocol of the fusion protocol stack uses reliable RDMA and TCP. When the data is received by the other end, the other end will reply ack for confirmation. The fusion protocol stack will call the underlying transmission protocol interface to query the underlying layer whether the data packet has been acked by the other end. Only the data packets that have been acked will be confirmed to be sent successfully to the application software to ensure reliable data transmission.

[0137] Data retransmission task: Although the underlying transmission protocol of the fusion protocol stack is reliable, data packet loss may still occur when the underlying data channel is abnormal. To ensure completely reliable data transmission, the fusion transmission stack has designed a retransmission mechanism on the public framework. When a data packet is delivered to the network, a timestamp will be added to the data packet. If the data packet has not been successfully sent for more than 5 seconds, it will be retransmitted and the timestamp will be updated. The data packet will not be cleared from the sending queue until it is successfully confirmed or the number of retransmissions reaches the upper limit due to a long-term network abnormality, and the sending failure will be returned to the application software.

[0138] Data receiving task: call the interface provided by the underlying protocol stack to receive data and submit the received data packets to the application software for processing.

[0139] Transmission protocol module; used for sending and receiving processing of each underlying protocol stack, that is, providing a unified API interface and calling each underlying protocol stack interface, where the protocol stack interface includes an RDMA protocol processing unit, a user-mode TCP protocol processing unit, and a kernel-mode TCP protocol processing module unit.

[0140] Embodiment 4:

[0141] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions implement the method of embodiment 1 or 2 when executed by the processor;

[0142] In practical applications, the processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller unit (MCU), a microprocessor or other electronic components to execute the methods in the above embodiments.

[0143] The method implemented in this embodiment is as described in Embodiment 1.

[0144] Embodiment five:

[0145] This embodiment further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method of the first or second embodiment is implemented;

[0146] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0147] The method implemented in this embodiment is as described in Embodiment 1 or 2.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A traffic communication and dynamic switching method based on three-stack fusion, characterized in that: include: S01, initializing the fusion protocol stack of each computing node and storage node; S02, according to the computing node, recording the time point t1 by sending a negotiation message, and recording the time point t2 according to the reply of the negotiation message; S03, judging whether the computing node and the storage node are in the same AZ by the round-trip delay between the time t1 and t2 according to the configured round-trip delay threshold; S04, when the computing node and the storage node belong to the same AZ, establishing a read and write request by integrating RDMA as a transmission protocol; S14: When the computing node and the storage node do not belong to the same AZ, a read / write request is established using user-mode TCP as a transmission protocol; S05, when an abnormality occurs in the transmission protocol of the computing node and the storage node, the computing node migrates the service data to the kernel TCP; S06, re-establishing the read and write request between the computing node and the storage node according to S04; and migrating the service data in the kernel TCP to the storage node according to the corresponding transmission type; Among them, S06 specifically includes: S061, reclaiming QP resources through the computing node and notifying the storage node to reclaim QP resources, and periodically checking the IB network card status; S062, when the network card status is restored, re-establishing the RDMA QP channel between the computing node and the storage node; S063: According to the RDMA QP channel, the service data is migrated back to the RDMA QP through the computing node.

2. According to claim 1, a traffic communication and dynamic switching method based on three-stack fusion is characterized in that: S04 specifically includes: S041, when the computing node and the storage node are in the same AZ and support integrated RDMA, select integrated RDMA as the transmission protocol of the computing node; S042, creating a QP according to the computing node, creating a QP in the storage node through a QP negotiation message, and completing the RDMA QP channel between the computing node and the storage node; S043: According to the RDMA QP channel, the computing node sends a read / write request to the storage node to implement service communication.

3. According to claim 2, a traffic communication and dynamic switching method based on three-stack fusion is characterized in that: S05 specifically includes: S051, the fusion protocol stack monitors that the IB device is abnormal, and the QP is in the error state, establishes a kernel-mode TCP connection through the computing node, and sends a TCP socket connection request to the storage node to complete the channel creation; S052, the computing node migrates the business data to the kernel TCP and maintains the read-write state.

4. According to claim 1, a traffic communication and dynamic switching method based on three-stack fusion is characterized in that: S14 specifically includes: S141, when the computing node and the storage node are in different AZs, and their RTBB is greater than the round-trip delay threshold and both support user-mode TCP, user-mode TCP is selected as the transmission protocol of the computing node; S142, creating a user-mode TCP socket according to the computing node, and initiating a link establishment request to the storage node, and completing the link establishment through the storage node replying to the link establishment request; S143, according to the link establishment, a read / write request is sent to the storage node through the computing node to realize business communication.

5. According to claim 4, a traffic communication and dynamic switching method based on three-stack fusion is characterized in that: S05 specifically includes: S151, when a user-mode TCP channel exception occurs in the computing node, the underlying protocol stack closes the user-mode TCP socket and creates a kernel-mode TCP socket, and sends a kernel-mode TCP socket connection request to the storage node to complete the channel creation; S152, the computing node migrates the business data to the kernel TCP and maintains the read-write state.

6. A traffic communication and dynamic switching method based on three-stack fusion according to claim 5, characterized in that: S06 is replaced with the following steps: S161, the computing node recreates a user-mode TCP socket and establishes a user-mode TCP channel with the storage node; S162: According to the user-mode TCP channel, the service data is migrated back to the user-mode TCP through the computing node.

7. A traffic communication and dynamic switching device based on three-stack fusion, used to execute the traffic communication and dynamic switching method based on three-stack fusion according to any one of claims 1 to 6, characterized in that: The device comprises: External interface module: used to provide a unified calling interface for application software; wherein, the application software is used to execute the traffic communication and dynamic switching method based on three-stack fusion according to any one of claims 1-6; A connection management module, used to implement the negotiation of control messages and the adaptive selection of transmission protocols between computing nodes and storage nodes; Task scheduling module, used to dynamically load and unload tasks as the transmission protocol changes; The task execution module is used to encapsulate the data processing in the protocol into tasks and add them to the task execution queue; Transmission protocol module; used for sending and receiving processing of each underlying protocol stack, including RDMA protocol processing unit, user-mode TCP protocol processing unit and kernel-mode TCP protocol processing module unit.

8. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the traffic communication and dynamic switching method based on three-stack fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the traffic communication and dynamic switching method based on three-stack fusion as described in any one of claims 1 to 6.

Images