Rdma packet forwarding method and network card

Abstract

The application relates to the communication technical field, discloses a RDMA message forwarding method and a network card, the method is applied to the network card containing a hardware processing module, is executed by the hardware processing module, and comprises the following steps: screening a target message of an RDMA type from a received message through a preconfigured message screening flow table; performing accurate matching on the target message and a forwarding entry in a preconfigured forwarding matching flow table in the hardware processing module; wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry comprises feature information used for message matching and a corresponding forwarding action; and performing the forwarding action corresponding to the forwarding entry matched successfully on the target message. Through the network card hardware processing module, message screening, matching and forwarding are performed, all RDMA messages can be directly forwarded by the hardware processing module, delay and overhead caused by software slow path learning are eliminated, and the forwarding performance of the RDMA message is improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an RDMA message forwarding method and a network interface card (NIC). Background Technology

[0002] Remote Direct Memory Access (RDMA) is widely used in high-performance business scenarios in data centers and clusters due to its low latency, high throughput, and lack of host CPU involvement. In multi-node cluster networks, there is a large amount of RDMA message exchange between multiple network endpoints within and between nodes, which places extremely high demands on the real-time performance, dynamic adaptability, and stability of message forwarding.

[0003] Currently, cluster RDMA packet forwarding generally employs a two-stage forwarding mechanism: software slow path learning and hardware fast path forwarding. For entirely new communication paths between the source and destination network endpoints within the cluster—paths where no packet exchange has occurred and no forwarding rules have been established—the first RDMA packet on that path needs to be parsed and the path learned (i.e., software slow path learning) by the host software layer. After learning is complete, a forwarding flow table recording packet matching characteristics and corresponding forwarding actions is generated and sent to the network interface card (NIC). Subsequent RDMA packets on this path can then be forwarded at high speed within the NIC, relying on the forwarding flow table (i.e., hardware fast path forwarding).

[0004] The above-mentioned two-level forwarding mechanism has a core technical flaw: for RDMA packets with a completely new communication path, they must go through the slow path of the host software layer to complete the parsing, learning and forwarding processing. This not only causes the first RDMA packet to have a large forwarding delay and unstable network latency, but also occupies host resources. It cannot achieve pure hardware high-speed forwarding of all RDMA packets, and it is difficult to meet the RDMA forwarding requirements of high-performance clusters with ultra-low latency and no computing power loss. Summary of the Invention

[0005] This application provides an RDMA packet forwarding method and network interface card (NIC). The NIC hardware processing module performs packet filtering, matching, and forwarding, enabling all RDMA packets to be directly forwarded by the hardware processing module. This eliminates the latency and overhead caused by software slow path learning and improves the forwarding performance of RDMA packets.

[0006] In a first aspect, one embodiment of this application provides an RDMA packet forwarding method, applied to a network interface card (NIC) including a hardware processing module, the method being executed by the hardware processing module, and including the following steps: The target packets of type RDMA are filtered out from the received packets using a pre-configured packet filtering flow table. The target packet is matched with the forwarding entries in the pre-configured forwarding matching flow table within the hardware processing module; wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry includes feature information for packet matching and the corresponding forwarding action; For the target message, perform the forwarding action corresponding to the successfully matched forwarding entry; The forwarding matching flow table is pre-configured before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the network card, so that all RDMA packets on the forwarding path corresponding to the target packet are directly forwarded through the hardware processing module.

[0007] Secondly, one embodiment of this application provides a network interface card (NIC), comprising: the NIC includes a hardware processing module; the hardware processing module includes: A pre-configured message filtering flow table is used to filter out target messages of type RDMA from received messages; A pre-configured forwarding matching flow table, wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry includes feature information for packet matching and the corresponding forwarding action; A matching unit is used to match the target packet with the forwarding entries in the forwarding matching flow table; The execution unit is used to perform the forwarding action corresponding to the successfully matched forwarding entry for the target packet; The forwarding matching flow table is pre-configured before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the network card, so that all RDMA packets on the forwarding path corresponding to the target packet are directly forwarded through the hardware processing module.

[0008] The RDMA packet forwarding method and network interface card (NIC) provided in this application embodiment execute all RDMA packet filtering, matching, and forwarding by the NIC hardware processing module, eliminating the need for software processing via the host control plane or NIC SoC system. Compared to the existing two-level forwarding mechanism where the first RDMA packet on a new forwarding path must traverse the slow software path, this embodiment pre-configures the forwarding matching flow table before the first RDMA packet arrives at the NIC. This allows RDMA packets on each forwarding path to be directly forwarded by the hardware processing module, completely eliminating the latency and overhead caused by the slow software path and significantly improving RDMA packet forwarding performance. Since the entire RDMA packet forwarding process is completed by the NIC's hardware logic, without involving software scheduling, context switching, or memory copying, the packet forwarding latency is extremely low and stable, while not consuming host resources, allowing the host to focus on business application processing and freeing up computing power. This embodiment can meet the stringent requirements for ultra-low latency and determinism in scenarios such as high-performance computing, distributed storage, and AI training. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the hardware architecture for an application scenario of the RDMA message forwarding method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating an embodiment of the RDMA message forwarding method provided in this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0012] The RDMA packet forwarding method and network interface card provided in this application are applicable to data center multi-node cluster scenarios and cloud-native virtualization scenarios.

[0013] In a multi-node cluster scenario in a data center, each node is equipped with the network interface card described in this application, and multiple network endpoints run on the nodes. These network endpoints need to perform a large amount of RDMA communication. Typical applications include distributed storage systems, high-performance computing clusters, and AI training clusters.

[0014] In cloud-native virtualization scenarios, Pods in container orchestration platforms such as Kubernetes obtain independent network channels by binding to VF, enabling high-speed RDMA communication between Pods. Typical applications include containerized microservices and cloud-native databases.

[0015] Figure 1 This is a schematic diagram of the hardware architecture for an application scenario of the RDMA message forwarding method provided in this application. Figure 1 As shown, the entire cluster consists of multiple nodes (node ​​1 to node n) and switches used to connect the multiple nodes.

[0016] Each node is a computing unit in the cluster, corresponding to a host. Each node includes: host software layer, Pod, VF (Virtual Function), and network interface card.

[0017] The host software layer runs on the node host and includes the operating system, host control plane, OK-CNI plugin, Kubelet, SR-IOV device plugin, etc. The host software layer is responsible for: Pod lifecycle management (creation, destruction, status monitoring), VF generation and allocation, configuration flow table distribution, and other tasks.

[0018] A Pod is a network endpoint instance running on a node. Multiple Pods run on each node (e.g., Pod_1_1 and Pod_1_2 on node 1, and Pod_n_1 and Pod_n_2 on node n). Each Pod obtains an independent channel for direct communication with the network interface card by binding to a VF assigned to it by the host (e.g., VF_1_1 and VF_1_2 on node 1, and VF_n_1 and VF_n_2 on node n).

[0019] The network interface card (NIC) is deployed on the nodes, with one NIC per node (node ​​1 corresponds to NIC 1, and node n corresponds to NIC n). The NIC in this application is a smart NIC, which internally includes a SoC (System on Chip), a hardware processing module, and physical ports.

[0020] The hardware processing module is a hardware logic unit inside the network card, used to perform packet filtering, matching, modification and forwarding operations without the need for software intervention.

[0021] A SoC (System on Chip) is an embedded processor system inside a network interface card (NIC). It mainly performs control functions such as flow table configuration, protocol processing, and network management, and works with hardware processing modules to complete packet forwarding.

[0022] The physical port is the physical interface between the network card and the outside world. It is connected to the switch via a network cable to enable the sending and receiving of cross-node packets.

[0023] The switch connects to the physical ports of the network cards of each node, enabling network communication between different nodes.

[0024] For ease of illustration, Figure 1 The example provided illustrates a Pod in a Kubernetes environment as a network endpoint. Those skilled in the art should understand that the network endpoint described in this application is not limited to a Pod, but can also be a container, virtual machine, or other communicable unit with an independent network identifier. Furthermore, the number of Pods in each node, the number of Virtual Functions (VFs) in each network interface card (NIC), and the number of nodes are not limited to... Figure 1 The specific number shown, Figure 1 This is for illustrative purposes only; in actual deployment, it can be flexibly configured according to business needs.

[0025] based on Figure 1 The hardware architecture shown in this application supports the following four RDMA packet forwarding scenarios: Scenario 1: Pods on this node sending messages to Pods on other nodes (cross-node outbound) A source Pod on this node sends an RDMA message to a destination Pod on another node. For example, Pod_1_1 on node 1 sends an RDMA message to Pod_n_2 on node n. The message originates from the source Pod, is processed by the network interface card (NIC) of this node, and then forwarded by the switch to the NIC of the destination node, finally reaching the destination Pod.

[0026] Scenario 2: Pods from other nodes send messages to Pods on this node (cross-node inbound) Source Pods on other nodes send RDMA packets to destination Pods on this node. For example, Pod_n_2 on node n sends an RDMA packet to Pod_1_1 on node 1. The packet originates from the source node, is forwarded by the switch to the network interface card (NIC) of this node, processed by the NIC, and then delivered to the destination Pod.

[0027] Scenario 3: Pod forwarding within the same node (inter-node forwarding) A source Pod on this node sends an RDMA packet to a destination Pod on the same node. For example, Pod_1_1 on node 1 sends an RDMA packet to Pod_1_2 on node 1. The packet originates from the source Pod, is processed by the network interface card of this node, and is directly forwarded to the destination Pod on the same node without going through a switch.

[0028] Scenario 4: Pod sends to itself (local loopback) A Pod sends an RDMA message to itself for testing scenarios such as device debugging and traffic self-testing. For example, Pod_1_1 on node 1 sends an RDMA message to itself. After the message is sent from the Pod, it is processed by the network interface card and then looped back directly to the source Pod.

[0029] This application provides an RDMA message forwarding method that can be applied to... Figure 1 The network interface card (NIC) shown demonstrates a method executed by its hardware processing module to achieve high-speed forwarding of RDMA packets. (See reference.) Figure 2 The method specifically includes the following steps: S201. Target packets of type RDMA are filtered from the received packets using a pre-configured packet filtering flow table.

[0030] The network interface card's hardware processing module receives packets from various VFs (corresponding to Pods on this node) or physical ports (corresponding to other nodes) within the node. The hardware processing module has a pre-configured packet filtering flow table, which is used to identify RDMA type packets.

[0031] It should be noted that before the network card performs packet forwarding, its internal packet filtering flow table has been pre-configured, and all RDMA packets (including the first RDMA packet) can be filtered in the hardware processing module.

[0032] In one specific embodiment, the message filtering flow table includes: an RDMA protocol characteristic field and a cluster network segment field.

[0033] The RDMA protocol feature field is used to filter messages that conform to the RDMA protocol characteristics. The specific content of the RDMA protocol feature field can be flexibly configured according to the actual RDMA protocol type used, and this application is not limited to this. In one example, for the RoCEv2 protocol, the RDMA protocol feature field includes the UDP protocol type and the destination port number (e.g., the default port 4791). In another example, for the InfiniBand protocol, the RDMA protocol feature field includes the InfiniBand network layer protocol type.

[0034] The cluster network segment field is used to filter packets from those conforming to the RDMA protocol characteristics, selecting those whose source IP address and / or destination IP address belong to a preset cluster network segment as target packets. The cluster network segment field is a preset set of IP address segments (such as 192.168.1.0 / 24, 10.0.0.0 / 8, etc.) used to identify the IP address range of network endpoints within the cluster, and can be pre-configured according to the network plan during cluster deployment.

[0035] In this application, the filtering conditions for inbound and outbound packets when filtering traffic using the cluster network segment field can be different, specifically: For incoming packets (with the destination network endpoint located on this node), the filtering criteria focus on whether the destination IP address belongs to the preset cluster network segment. That is, if the destination IP address of the incoming packet belongs to the preset cluster network segment, then the packet is regarded as the target packet.

[0036] For outbound packets (the source network endpoint is located on this node), the filtering criteria consider whether the source IP address and the destination IP address belong to the preset cluster network segment. That is, if both the source IP address and the destination IP address of the outbound packet belong to the preset cluster network segment, then the packet is regarded as the target packet.

[0037] The filtering and judgment of inbound and outbound messages are performed in parallel, without any order.

[0038] The source IP address and destination IP address of a packet forwarded within a node (both the source and destination network endpoints are located on this node) both belong to the preset cluster network segment. Therefore, it meets the filtering conditions for both outbound and inbound packets and will also be identified as the target packet.

[0039] It should be noted that the RDMA protocol characteristic field and the cluster network segment field can be integrated into the same flow table, or they can be deployed in multiple independent flow tables (such as the first filtering flow table and the second filtering flow table). This application does not impose any restrictions on this. As long as the packet filtering function can be implemented, it is acceptable.

[0040] The hardware processing module extracts information such as the protocol type, port number, source IP address, and destination IP address of the received packet and matches it with the rules in the packet filtering flow table. When a packet simultaneously meets the RDMA protocol characteristics and the cluster network segment conditions, the packet is identified as a target packet and enters the subsequent forwarding matching flow table processing procedure (i.e., step S202); otherwise, it is processed according to the conventional forwarding method (such as taking the software forwarding path or ordinary hardware forwarding), which is not limited in this application.

[0041] The packet filtering flow table is pre-configured by the host control plane according to the cluster network plan and distributed to the hardware processing module during network interface card (NIC) initialization or cluster network configuration changes. Specifically, the host control plane generates configuration rules for the packet filtering flow table based on the RDMA protocol type (e.g., RoCEv2) and cluster network segment (e.g., container network segment, Pod network segment) planned during cluster deployment, and distributes these rules to the NIC's hardware processing module via the driver. The hardware processing module receives and stores these rules, completing the pre-configuration of the packet filtering flow table.

[0042] It should be noted that the configuration of the packet filtering flow table is completed before the network endpoint begins communication. Therefore, when the first RDMA packet arrives, the corresponding filtering rules already exist in the hardware processing module.

[0043] S202. Match the target packet with the forwarding entries in the pre-configured forwarding matching flow table within the hardware processing module.

[0044] The hardware processing module also has a pre-configured forwarding matching flow table. Each forwarding entry in this flow table uniquely corresponds to a forwarding path, and each entry includes feature information for packet matching and the corresponding forwarding action.

[0045] In this application, a forwarding path refers to a communication channel from a source network endpoint to a destination network endpoint. Each forwarding path has a unique directionality, i.e., from the source network endpoint to the destination network endpoint. Bidirectional communication between the same pair of network endpoints corresponds to two independent forwarding paths, for example, one forwarding path is from network endpoint A to network endpoint B, and the other is from network endpoint B to network endpoint A.

[0046] The characteristic information in a forwarding entry is used to uniquely identify a forwarding path. Specifically, the characteristic information includes a combination of fields that uniquely identify the source network endpoint and the destination network endpoint. For example, the characteristic information can be a combination of fields such as source network endpoint identifier, destination network endpoint identifier, a five-tuple (source IP, destination IP, source port, destination port, protocol type) or a seven-tuple (further including source MAC, destination MAC). The combination of fields in the characteristic information must be able to uniquely identify a forwarding path.

[0047] The forwarding action in a forwarding entry refers to the operation instruction that needs to be performed on the target packet. When the characteristic information of the target packet is precisely matched with the characteristic information of the forwarding entry, the hardware processing module executes the forwarding action, processes the packet accordingly (such as modifying the address, encapsulating a tunnel, specifying the output port, etc.), and forwards it.

[0048] It should be noted that before the network interface card (NIC) performs packet forwarding, its internal forwarding matching flow table has already been pre-configured to ensure that all RDMA packets (including the first RDMA packet) can be accurately matched in the hardware processing module. Specifically, the host control plane can monitor events of network endpoints within the cluster. When a new network endpoint is detected in the cluster, the corresponding forwarding entry is configured. Before the first RDMA packet on the forwarding path of that network endpoint arrives at the NIC, the configured forwarding entry is sent to the NIC's hardware processing module to configure the forwarding matching flow table. This ensures that before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the NIC, the NIC's forwarding matching flow table already contains the forwarding entry for that forwarding path, so that all RDMA packets on that forwarding path are directly forwarded through the hardware processing module without needing to be sent to the host software layer.

[0049] The hardware processing module extracts the characteristic information of the target packet (such as source MAC, destination MAC, source IP, destination IP, source port, destination port, protocol type, etc.) and performs an exact match (i.e., a complete equality match) with the characteristic information of each forwarding entry in the forwarding matching flow table. If the match is successful, the forwarding action corresponding to the forwarding entry is obtained; if the match fails, the target packet is sent to the host control plane for processing.

[0050] S203. Perform the forwarding action corresponding to the successfully matched forwarding entry for the target message.

[0051] The hardware processing module processes the target packet according to the forwarding action in the successfully matched forwarding entry. The specific type of forwarding action (such as tunnel encapsulation, tunnel decapsulation, MAC address modification, output port specification, local loopback, etc.) is determined by the matched forwarding entry, and this application does not limit this. After the forwarding action is executed, the packet is sent to the specified output port (physical port or VF), completing the forwarding processing of the RDMA packet.

[0052] The RDMA packet forwarding method provided in this embodiment performs all RDMA packet filtering, matching, and forwarding by the network interface card (NIC) hardware processing module, without requiring software processing by the host control plane or NIC SoC system. Compared to the existing two-level forwarding mechanism where the first RDMA packet on a new forwarding path must take the software slow path, the forwarding matching flow table in this embodiment is pre-configured before the first RDMA packet arrives at the NIC. This allows RDMA packets on each forwarding path to be directly forwarded by the hardware processing module, completely eliminating the latency and overhead caused by the software slow path and significantly improving the forwarding performance of RDMA packets.

[0053] Since the entire RDMA packet forwarding process is completed by the network card's hardware logic, without involving software scheduling, context switching, memory copying, or other operations, the packet forwarding latency is extremely low and stable. Simultaneously, it does not consume host resources, allowing the host to focus on business application processing and freeing up computing power. This embodiment can meet the stringent requirements for ultra-low latency and determinism in scenarios such as high-performance computing, distributed storage, and AI training.

[0054] This embodiment further illustrates the configuration method of the forwarding matching flow table based on the previous embodiments. The configuration of the forwarding matching flow table is performed by the node's host control plane. Specifically, it includes the following steps: S301, listen for events on each network endpoint within the cluster.

[0055] The host control plane (including components such as the OK-CNI plugin and Kubelet) continuously listens for events on network endpoints within the cluster, including but not limited to: creation, deletion, status changes (such as changing from not ready to ready), and IP address changes of network endpoints.

[0056] In this application, the ready state refers to the network endpoint having completed the hardware configuration for forwarding matching flow tables and possessing the prerequisites for sending and receiving RDMA messages (i.e., the hardware forwarding path is in place), but at this time, the network endpoint has not yet started sending and receiving RDMA messages. After entering the ready state, the network endpoint still needs to go through the RDMA connection establishment process before it can formally enter the communication phase and start sending and receiving messages.

[0057] S302. In response to the detection of a new network endpoint being created, configure the forwarding entry corresponding to the new network endpoint.

[0058] Each node's host control plane in the cluster listens for events from all network endpoints across all nodes. Upon detecting the creation of a new network endpoint, each node's host control plane configures a forwarding entry related to its own node (i.e., a forwarding entry between the node's own network endpoint and the new network endpoint) and sends it to the hardware processing module of the node's network interface card.

[0059] In one specific embodiment, when the host control plane of a node detects an event indicating the creation of a new network endpoint (which can be any network endpoint in the cluster), it first determines whether the newly created network endpoint belongs to this node (the node corresponding to this host): If it is determined that the newly created network endpoint is located on this node, then configure forwarding entries from the newly created network endpoint to all ready network endpoints in the cluster, and forwarding entries from all ready network endpoints in the cluster to the newly created network endpoint; and after the forwarding entries are distributed, set the newly created network endpoint to the ready state. If it is determined that the newly created network endpoint is located on another node in the cluster (i.e., not this node), then wait for the new network endpoint to enter the ready state before configuring forwarding entries from the newly created network endpoint to the ready network endpoint on this node, and forwarding entries from the ready network endpoint on this node to the newly created network endpoint.

[0060] Among them, a ready network endpoint refers to a network endpoint that has entered the ready state.

[0061] It should be noted that the host control plane of this node is only responsible for configuring forwarding entries related to this node, namely: forwarding entries between network endpoints within this node, and forwarding entries between network endpoints of this node and network endpoints of other nodes. Forwarding entries within other nodes are configured by the host control plane of the respective node.

[0062] With the above configuration, the bidirectional forwarding path between the new network endpoint and all ready network endpoints in the cluster is automatically established as soon as the new network endpoint is created, and the corresponding forwarding entry is sent to the forwarding matching flow table of the network card hardware processing module, thereby realizing the instant pre-configuration and hardware unloading of the forwarding path.

[0063] S303. Before a new network endpoint begins sending and receiving packets, the configured forwarding entries are sent to the hardware processing module to configure the forwarding matching flow table.

[0064] The host control plane sends the configured forwarding entries to the network card's hardware processing module, which then writes the forwarding entries into the forwarding matching flow table, completing the flow table configuration.

[0065] It should be noted that the above configuration steps are executed immediately after the new network endpoint is created, and completed before the network endpoint begins sending and receiving packets. The newly created network endpoint enters a ready state after the forwarding matching flow table configuration is complete. At this time, its hardware forwarding path is in place, but it has not yet started sending and receiving packets. The newly created network endpoint can only establish an RDMA connection and begin sending and receiving packets after entering the ready state. Therefore, when the first RDMA packet on the forwarding path of this network endpoint arrives at the network card, the forwarding matching flow table of the hardware processing module already contains a forwarding entry corresponding to the newly created network endpoint. It can directly perform accurate matching and forwarding in the network card hardware processing module without needing to send it to the host control plane to trigger learning.

[0066] For other nodes in the cluster, after the host control plane of each node listens for the creation event of the network endpoint, it waits for the network endpoint to enter the ready state before configuring the forwarding entry from its own endpoint to that endpoint. Since the newly created network endpoint still needs to go through the RDMA connection establishment process after entering the ready state before it can start sending and receiving packets, and this connection establishment process provides a sufficient configuration time window, the forwarding entries of other nodes can also be configured and sent to the corresponding network interface card before the first RDMA packet arrives.

[0067] by Figure 1 Using the architecture shown as an example, the configuration process of the forwarding matching flow table will be explained.

[0068] When a new Pod_1_1 is created on node 1, the host control plane of node 1 performs the following operations: establishes bidirectional forwarding entries between Pod_1_1 and other Pods on node 1 (such as Pod_1_2); establishes bidirectional forwarding entries between Pod_1_1 and each Pod (such as Pod_n_1, Pod_n_2) on other nodes (such as node n); distributes the established forwarding entries to the hardware processing module in the network interface card of node 1; and sets Pod_1_1 to the ready state.

[0069] When other nodes (such as node n) also synchronously detect that Pod_1_1 has been created and entered the ready state, they perform the following operations: establish bidirectional forwarding entries between each Pod (such as Pod_n_1, Pod_n_2) on node n and Pod_1_1; and send the established forwarding entries to the hardware processing module in the network card of node n.

[0070] Through the above mechanism, Pod_1_1 establishes a bidirectional forwarding path with all ready Pods in the cluster immediately after its creation, and the forwarding entries are all distributed to the hardware processing modules of the network interface cards (NICs) of each node. When any Pod sends its first RDMA packet to another Pod, the corresponding forwarding entry already exists in the NIC's hardware processing module, allowing for direct hardware forwarding.

[0071] It should be noted that when a network endpoint is deleted or its IP address changes, the host control plane will also listen for the corresponding event and perform the appropriate processing operations: In response to a deletion event, the host control plane deletes all forwarding entries associated with that network endpoint and notifies the host control planes of other nodes to delete them synchronously, thus preventing invalid forwarding entries from consuming hardware resources.

[0072] In response to IP address change events, the host control plane updates the characteristic information (such as source IP, destination IP, etc.) in the forwarding entries related to the network endpoint and resends them to the hardware processing module to ensure the accuracy of the forwarding path.

[0073] This application does not specify the handling method for events such as deletion and IP address change, as long as it can ensure that the entries in the forwarding matching flow table are synchronized with the actual state of the network endpoints within the cluster.

[0074] In this embodiment, the host control plane of each node in the cluster listens for events of all network endpoints within the cluster. When a new network endpoint is detected, the host control plane of each node configures a forwarding entry related to its own node and sends it to the hardware processing module of its network interface card, thus realizing the proactive pre-configuration of the forwarding matching flow table. Compared with the passive configuration method of learning triggered by the first RDMA packet in the prior art, in this embodiment, all RDMA packets (including the first RDMA packet) can be directly forwarded by the hardware processing module, completely skipping the software slow path forwarding process and eliminating the latency and overhead caused by the software slow path.

[0075] Furthermore, existing flow tables mostly operate in a passive learning or fixed configuration mode, unable to generate and update forwarding entries in real time based on dynamic events such as network endpoint online / offline status changes within the cluster. This results in poor dynamic adaptability and is prone to issues such as packet forwarding failures and packet loss. This embodiment, however, monitors network endpoint events in real time through the host control plane, enabling it to generate and update forwarding entries in real time based on dynamic changes in endpoints. This provides strong dynamic adaptability and avoids forwarding failures and packet loss caused by untimely configuration.

[0076] This embodiment further refines the forwarding action types corresponding to the forwarding entries in the forwarding matching flow table.

[0077] In order to transmit RDMA messages from one node to other nodes in the cluster, the original RDMA messages need to be encapsulated into a format suitable for the underlying network transmission, such as tunnel encapsulation formats like VXLAN and Geneve. This application does not limit the specific encapsulation method.

[0078] The forwarding action corresponding to the forwarding entry in the forwarding matching flow table is any of the following types: (1) First type of forwarding action: Perform tunnel encapsulation, and modify the source MAC address in the outer tunnel header to the MAC address of the local node's network card and the destination MAC address to the MAC address of the peer node's network card, and forward the packet according to the source MAC address and destination MAC address in the outer tunnel header.

[0079] The first type of forwarding action applies to scenarios where the local node's network endpoint forwards packets to other nodes' network endpoints in the cluster (i.e., Scenario 1). Specific operations include: performing tunnel encapsulation on the original RDMA packet, adding an outer tunnel header (such as VXLAN, Geneve, etc.); modifying the source MAC address in the outer tunnel header to the local node's network card MAC address, and modifying the destination MAC address in the outer tunnel header to the peer node's network card MAC address; and forwarding the encapsulated packet to the peer node based on the modified source and destination MAC addresses in the outer tunnel header.

[0080] The MAC address in the original RDMA message remains unchanged to ensure that the network card of the destination node can correctly forward it to the destination network endpoint based on the MAC address in the original RDMA message.

[0081] by Figure 1 For example, when Pod_1_1 on node 1 sends an RDMA packet to Pod_n_1 on node n, the hardware processing module of network interface card 1 first identifies it as an RDMA packet through the packet filtering flow table. Then, it performs a precise match between the RDMA packet and the feature information in each forwarding entry in the forwarding matching flow table. After determining the matching forwarding entry, it executes the forwarding action in that entry: it encapsulates the original RDMA packet into a tunnel, modifies the source MAC address in the outer tunnel header to the MAC address of network interface card 1, modifies the destination MAC address to the MAC address of network interface card n, and then sends it to the switch through the physical port of network interface card 1. The switch then forwards the packet to node n.

[0082] Link aggregation groups are a technology that binds multiple physical ports into a single logical link to improve bandwidth and achieve link redundancy. In one optional embodiment, when the local network interface card (NIC) is configured with a link aggregation group, a link aggregation group identifier is also configured in the first type of forwarding action. When the hardware processing module executes the first type of forwarding action, it directly sends the packet to the link aggregation group according to the configured link aggregation group identifier. The link aggregation group then automatically selects one of the physical ports, Port0 or Port1, to complete the actual transmission of the packet based on a load balancing strategy (such as hash calculation based on the source MAC, destination MAC, source IP, and destination IP in the outer tunnel header). The load balancing strategy and the number of physical ports of the link aggregation group can be flexibly configured according to actual deployment requirements, and this application does not impose any limitations on this.

[0083] It should be noted that the MAC address of the local node's network card, the MAC address of the peer node's network card, and the link aggregation group identifier in the forwarding action are all predetermined based on the corresponding forwarding path. When a forwarding entry is configured, the host control plane pre-fills in the relevant information based on the corresponding forwarding path when configuring the forwarding entry, and sends it to the hardware processing module along with the forwarding entry. The hardware processing module directly uses these pre-configured parameters when executing the forwarding action, without the need for dynamic learning or calculation.

[0084] (2) Second type of forwarding action: forward the packet to the corresponding network endpoint according to the preset port identifier.

[0085] The port identifier in the forwarding action is predetermined based on the corresponding forwarding path, specifically determined by the destination network endpoint corresponding to that forwarding path. When a forwarding entry is configured, the host control plane writes the corresponding port identifier into the forwarding action based on the destination network endpoint information (such as the node it resides in, the associated VF, etc.). The hardware processing module then directly uses this port identifier to complete packet delivery.

[0086] The second type of forwarding action is applicable to scenarios where the network endpoint of any node in the cluster (including the current node) forwards packets to the network endpoint of the current node (i.e., scenarios two, three, and four), because the core of these scenarios is to deliver packets to the destination network endpoint based on the port identifier.

[0087] Specifically, when the local network card receives a tunnel encapsulated packet from another node, the hardware processing module first performs tunnel decapsulation, stripping the outer tunnel header to obtain the original packet. Then, it identifies the original packet as an RDMA packet through the packet filtering flow table. Next, it performs precise matching of the original packet with the feature information in each forwarding entry in the forwarding matching flow table. After determining the matching forwarding entry, it executes the forwarding action in that forwarding entry, that is, forwards the original packet to the corresponding network endpoint according to the port identifier in the forwarding action.

[0088] It should be noted that RDMA packets sent between different network endpoints within this node, as well as RDMA packets sent from a network endpoint within this node to itself, do not require encapsulation and decapsulation processing. Therefore, when the network interface card of this node receives a packet from a network endpoint of this node, the hardware processing module directly filters it through the packet filtering flow table, performs precise matching through the forwarding matching flow table, and executes the forwarding action after hitting the corresponding forwarding entry.

[0089] by Figure 1For example, when Pod_1_1 on node 1 sends an RDMA message to Pod_n_1 on node n, network interface n receives the message from the physical port. Since it is an RDMA message sent by another node, it needs to be tunneled and decapsulated first to obtain the original RDMA message. Then, the hardware processing module identifies it as an RDMA message through the message filtering flow table, and then performs a precise match between the RDMA message and the feature information in each forwarding entry in the forwarding matching flow table. After determining the matching forwarding entry (corresponding to the forwarding path Pod_1_1→Pod_n_1), the forwarding action in the forwarding entry is executed: according to the port identifier recorded in the forwarding action (the port identifier of VF_n_1), the RDMA message is sent to the VF_n_1 corresponding to Pod_n_1.

[0090] When Pod_1_1 on node 1 sends an RDMA packet to Pod_1_2 on the same node, the hardware processing module of network interface card 1 first identifies it as an RDMA packet through the packet filtering flow table, and then performs a precise match according to the forwarding matching flow table. After hitting the forwarding entry corresponding to the forwarding path Pod_1_1→Pod_1_2, it executes the forwarding action: according to the port identifier recorded in the forwarding action (the port identifier of VF_1_2), the RDMA packet is sent to the VF_1_2 corresponding to Pod_1_2.

[0091] When Pod_1_1 on node 1 sends an RDMA packet to itself, the hardware processing module of network interface card 1 performs packet filtering flow table filtering and forwarding matching flow table precise matching. After hitting the corresponding forwarding entry, it executes the forwarding action: according to the port identifier recorded in the forwarding action (the port identifier of VF_1_1), it sends the RDMA packet to the VF_1_1 corresponding to Pod_1_1.

[0092] It should be noted that a forwarding entry contains only one of the two types of forwarding actions mentioned above.

[0093] This embodiment categorizes forwarding actions into two types, adapting to four scenarios: cross-node outbound, cross-node inbound, intra-node forwarding, and local loopback. This allows the hardware processing module to execute corresponding operations based on different forwarding paths, achieving full-scenario hardware forwarding coverage for RDMA packets. Furthermore, operations such as tunnel encapsulation and MAC address modification are all completed within the hardware processing module, requiring no software intervention, further improving forwarding performance.

[0094] This embodiment, based on the aforementioned embodiments, further illustrates the configuration and uses of the general forwarding flow table.

[0095] In this embodiment, the hardware processing module is also pre-configured with a general forwarding flow table. When the forwarding matching flow table is not enabled, or is enabled but no corresponding forwarding entry is found, the hardware processing module forwards the packet using the general forwarding flow table.

[0096] The general forwarding flow table can be configured with various forwarding entries, including but not limited to forwarding entries corresponding to local loopback packets. Among them, local loopback packets refer to packets sent from a network endpoint with a destination address of the same network endpoint itself (e.g., packets sent by Pod_1_1 to itself), which are commonly used in testing scenarios such as device debugging, traffic self-testing, and functional integration testing.

[0097] In one example, the general forwarding flow table may include forwarding entries from each network endpoint on the node where the network interface card resides to itself. When a network endpoint sends an RDMA (Local Loopback) packet to itself, the hardware processing module matches the corresponding loopback forwarding entry in the general forwarding flow table and directly loops the packet back to the source network endpoint itself. For example, if node 1 contains Pod_1_1 and Pod_1_2, then the general forwarding flow table may contain one forwarding entry from Pod_1_1 to Pod_1_1 itself, and one forwarding entry from Pod_1_2 to Pod_1_2 itself.

[0098] The general forwarding flow table is pre-configured by the host control plane and distributed to the hardware processing module, and its forwarding entries are relatively fixed.

[0099] It should be noted that the general forwarding flow table has a lower priority than the forwarding matching flow table. When processing packets, the hardware processing module prioritizes matching the forwarding matching flow table; if the forwarding matching flow table is not enabled or does not match the corresponding forwarding entry, it further matches the general forwarding flow table. The general forwarding flow table serves as a fallback forwarding path, ensuring that packets can still be forwarded normally when the forwarding matching flow table is not configured or does not match.

[0100] This application embodiment also provides a network interface card (NIC), the NIC including a hardware processing module, the hardware processing module including: A pre-configured message filtering flow table is used to filter out target messages of type RDMA from received messages; A pre-configured forwarding matching flow table, wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry includes feature information for packet matching and the corresponding forwarding action; A matching unit is used to match the target packet with the forwarding entries in the forwarding matching flow table; The execution unit is used to perform the forwarding action corresponding to the successfully matched forwarding entry for the target packet; The forwarding matching flow table is pre-configured before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the network card, so that all RDMA packets on the forwarding path corresponding to the target packet are directly forwarded through the hardware processing module.

[0101] In one optional embodiment, the network interface card (NIC) is located on a node in the cluster, and each node contains at least one network endpoint; the host control plane of the node where the NIC is located is used for: Listen for events at each network endpoint within the cluster; In response to the detection of a new network endpoint being created, configure the forwarding entry corresponding to the new network endpoint; Before the new network endpoint begins sending and receiving packets, the configured forwarding entries are sent to the hardware processing module to configure the forwarding matching flow table.

[0102] Furthermore, configuring the forwarding entry corresponding to the new network endpoint specifically includes: If the new network endpoint is located on the node where the network card is located, then configure forwarding entries from the new network endpoint to each ready network endpoint in the cluster, and forwarding entries from each ready network endpoint in the cluster to the new network endpoint, and after completing the distribution of forwarding entries, set the new network endpoint to the ready state. If the new network endpoint is located on another node in the cluster, then after the new network endpoint enters the ready state, configure forwarding entries from each ready network endpoint on the node where the network card is located to the new network endpoint, and from the new network endpoint to each ready network endpoint on the node where the network card is located.

[0103] In one alternative embodiment, the network interface card is located on a node in the cluster, and each node contains at least one network endpoint; The forwarding action corresponding to the forwarding entry in the forwarding matching flow table is any of the following types: The first type of forwarding action is to perform tunnel encapsulation, modify the source MAC address in the outer tunnel header to the MAC address of the local node's network card, and modify the destination MAC address to the MAC address of the peer node's network card. The packet is forwarded based on the source MAC address and destination MAC address in the outer tunnel header, adapting to the scenario where the local node's network endpoint forwards packets to the network endpoints of other nodes in the cluster. The second type of forwarding action: forwards the packet to the corresponding network endpoint according to the preset port identifier, adapting to the scenario where the network endpoint of any node in the cluster forwards the packet to the network endpoint of this node.

[0104] In an optional embodiment, the hardware processing module is also pre-configured with a general forwarding flow table; the general forwarding flow table is used to process packet forwarding when the forwarding matching flow table is not enabled or no forwarding entry is matched.

[0105] The network card provided in this application embodiment adopts the same inventive concept as the above-described RDMA packet forwarding method. The implementation of this network card can refer to the implementation of the method and can achieve the same beneficial effects. Repeated parts will not be described again.

[0106] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as a limitation on the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for forwarding RDMA messages, characterized in that, Applied to a network interface card (NIC) containing a hardware processing module, the method is executed by the hardware processing module and includes the following steps: The target packets of type RDMA are filtered out from the received packets using a pre-configured packet filtering flow table. The target packet is matched with the forwarding entries in the pre-configured forwarding matching flow table within the hardware processing module; wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry includes feature information for packet matching and the corresponding forwarding action; For the target message, perform the forwarding action corresponding to the successfully matched forwarding entry; The forwarding matching flow table is pre-configured before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the network card, so that all RDMA packets on the forwarding path corresponding to the target packet are directly forwarded through the hardware processing module. The network interface card is located on one node in the cluster, and each node contains at least one network endpoint. The method also includes the following steps being performed on the host control plane of the node where the network interface card is located: Listen for events at each network endpoint within the cluster; In response to the detection of a new network endpoint being created, configure the forwarding entry corresponding to the new network endpoint; Before the new network endpoint begins sending and receiving packets, the configured forwarding entries are sent to the hardware processing module to configure the forwarding matching flow table.

2. The method according to claim 1, characterized in that, The configuration of the forwarding entries corresponding to the new network endpoint specifically includes: If the new network endpoint is located on the node where the network card is located, then configure forwarding entries from the new network endpoint to each ready network endpoint in the cluster, and forwarding entries from each ready network endpoint in the cluster to the new network endpoint, and after completing the distribution of forwarding entries, set the new network endpoint to the ready state. If the new network endpoint is located on another node in the cluster, then after the new network endpoint enters the ready state, configure forwarding entries from each ready network endpoint on the node where the network card is located to the new network endpoint, and from the new network endpoint to each ready network endpoint on the node where the network card is located.

3. The method according to claim 1, characterized in that, The network interface card is located on one node in the cluster, and each node contains at least one network endpoint. The forwarding action corresponding to the forwarding entry in the forwarding matching flow table is any of the following types: The first type of forwarding action is to perform tunnel encapsulation, modify the source MAC address in the outer tunnel header to the MAC address of the local node's network card, and modify the destination MAC address to the MAC address of the peer node's network card. The packet is forwarded based on the source MAC address and destination MAC address in the outer tunnel header, adapting to the scenario where the local node's network endpoint forwards packets to the network endpoints of other nodes in the cluster. The second type of forwarding action: forwards the packet to the corresponding network endpoint according to the preset port identifier, adapting to the scenario where the network endpoint of any node in the cluster forwards the packet to the network endpoint of this node.

4. The method according to claim 1, characterized in that, The hardware processing module is also pre-configured with a general forwarding flow table; the general forwarding flow table is used to process packet forwarding when the forwarding matching flow table is not enabled or no forwarding entry is matched.

5. The method according to any one of claims 1 to 4, characterized in that, The feature information includes a combination of fields used to uniquely identify the source network endpoint and the destination network endpoint.

6. The method according to any one of claims 1 to 4, characterized in that, The message filtering flow table includes: The RDMA protocol feature field is used to filter out messages that match the RDMA protocol features. The cluster network segment field is used to filter out packets whose source IP address and / or destination IP address belong to a preset cluster network segment from the packets that conform to the RDMA protocol characteristics, and use them as the target packets.

7. A network interface card (NIC), characterized in that, The network interface card (NIC) includes a hardware processing module, which includes: A pre-configured message filtering flow table is used to filter out target messages of type RDMA from received messages; A pre-configured forwarding matching flow table, wherein each forwarding entry in the forwarding matching flow table uniquely corresponds to a forwarding path, and each forwarding entry includes feature information for packet matching and the corresponding forwarding action; A matching unit is used to match the target packet with the forwarding entries in the forwarding matching flow table; The execution unit is used to perform the forwarding action corresponding to the successfully matched forwarding entry for the target packet; The forwarding matching flow table is pre-configured before the first RDMA packet on the forwarding path corresponding to the target packet arrives at the network card, so that all RDMA packets on the forwarding path corresponding to the target packet are directly forwarded through the hardware processing module. The network interface card is located on one node in the cluster, and each node contains at least one network endpoint. The host control plane of the node where the network interface card is located is used for: Listen for events at each network endpoint within the cluster; In response to the detection of a new network endpoint being created, configure the forwarding entry corresponding to the new network endpoint; Before the new network endpoint begins sending and receiving packets, the configured forwarding entries are sent to the hardware processing module to configure the forwarding matching flow table.

8. The network interface card according to claim 7, characterized in that, The network interface card is located on one node in the cluster, and each node contains at least one network endpoint. The forwarding action corresponding to the forwarding entry in the forwarding matching flow table is any of the following types: The first type of forwarding action is to perform tunnel encapsulation, modify the source MAC address in the outer tunnel header to the MAC address of the local node's network card, and modify the destination MAC address to the MAC address of the peer node's network card. The packet is forwarded based on the source MAC address and destination MAC address in the outer tunnel header, adapting to the scenario where the local node's network endpoint forwards packets to the network endpoints of other nodes in the cluster. The second type of forwarding action: forwards the packet to the corresponding network endpoint according to the preset port identifier, adapting to the scenario where the network endpoint of any node in the cluster forwards the packet to the network endpoint of this node.

Images