Novel packet-by-packet load balancing method, device and equipment and storage medium

Abstract

The invention discloses a novel packet-by-packet load balancing method and device, equipment and a storage medium. The method comprises the steps that network side equipment receives data packets; a target field and quintuple information in the data packet are obtained, the target field is used for identifying a data stream segment to which the data packet belongs, and the quintuple information comprises a source IP address, a target IP address, a source port number, a target port number and a transport layer protocol; performing hash calculation according to a target field and the quintuple information to obtain a hash result; and determining an output port of the data packet according to the hash result, so that the data packets in the same data stream segment are transmitted through a path corresponding to the same output port. According to the invention, the problem of congestion caused by uneven load in a high-performance computing network is solved, and meanwhile, the pressure on the out-of-order rearrangement capability of the network card at the receiving end is reduced.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a novel packet-by-packet load balancing method, apparatus, device, and storage medium. Background Technology

[0002] In data center networks and high-speed communication systems, load balancing technology is a key means to optimize network resource utilization and avoid congestion. Existing technologies mainly rely on flow-based load balancing methods, which perform hash calculations based on the five-tuple information of data packets (source IP address, destination IP address, source port number, destination port number, and transport layer protocol) to determine the forwarding path of the data packets. For example, existing technologies disclose a multi-path transmission scheme based on five-tuple hashing: the switch performs hash calculations on the five-tuples of data packets and selects the outgoing port, ensuring that data packets with the same five-tuple are transmitted through the same path, thereby achieving load balancing between different data flows. While this method is simple and easy to implement, it has significant drawbacks: hash calculations may be uneven, leading to hash polarization, where some paths are overloaded while others are idle, resulting in wasted network resources; simultaneously, data flows with the same five-tuple always pass through the same path, and when that path becomes congested, the performance of the entire data flow degrades, failing to dynamically adapt to network changes. In high-speed network environments (such as RoCEv2), this flow-by-flow load balancing method is prone to packet out-of-order delivery and congestion, because packets in the same data stream may become out of order due to path switching, requiring a retransmission mechanism and increasing overhead.

[0003] Another existing technology introduces the concept of Flowlets, which divides packets into micro-flows by detecting the time intervals between them, enabling finer-grained load balancing. This method identifies Flowlets on the switch side based on a fixed time threshold, thus allocating packets within the same Flowlet to the same path. However, this method relies on a fixed time interval, making it difficult to adapt to dynamically changing network loads: if the time interval is set too small, the number of Flowlets increases, raising the risk of out-of-order packets; if the time interval is set too large, the load balancing granularity is coarse, resulting in insignificant performance. Furthermore, it does not address how to utilize packet-specific fields (such as the PSN Sequence ID of RoCE packets or the header identification of IP packets) to optimize hash calculations, thus failing to solve the problem of insufficient path selection precision.

[0004] Existing technologies focus on Network Address Translation (NAT) and Port Translation, allocating network resources through hash-based load balancing. However, this approach primarily addresses address resource allocation and does not address packet path selection or load balancing optimization, particularly lacking support for high-performance network protocols such as RDMA (Remote Direct Memory Access). In real-world applications, with the surge in traffic driven by cloud computing, big data, and AI, network load exhibits burstiness and heterogeneity. Existing technologies cannot effectively combine packet-specific identifiers (such as protocol-specific fields) with path selection mechanisms, leading to uneven load distribution, low bandwidth utilization, and frequent congestion. Current technologies have not effectively integrated multiple elements, making it impossible to simultaneously solve uneven load distribution and congestion problems in high-performance network scenarios. Therefore, a new method is urgently needed that can perform load balancing on a packet-by-packet basis, incorporate packet-specific fields, and support dynamic path selection. Summary of the Invention

[0005] This application addresses some of the shortcomings mentioned in the background art by providing a novel packet-by-packet load balancing method, apparatus, device, and storage medium.

[0006] In a first aspect, this application provides a novel per-packet load balancing method, comprising: Network-side equipment receives data packets; Obtain the target field and 5-tuple information from the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the 5-tuple information includes the source IP address, destination IP address, source port number, destination port number, and transport layer protocol; A hash calculation is performed based on the target field and the quintuple information to obtain the hash result; Based on the hash result, the outgoing port of the data packet is determined, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0007] In one embodiment of the first aspect, if the target field is a sequence number sequence identifier of a remote direct memory access packet on converged Ethernet, then a preset number of bits is taken from the sequence number sequence identifier to participate in the hash calculation; If the target field is the header identifier field of the network layer core protocol data packet, then a preset number of bits are taken from the header identifier field to participate in the hash calculation.

[0008] This embodiment breaks hash polarization by using a more granular "segment-by-segment" load balancing approach than "flow-by-flow," enabling traffic to be distributed more evenly across multiple network paths, thus improving network bandwidth utilization and overall performance. Compared to pure packet-by-packet load balancing, this embodiment bundles consecutive data packets together for transmission, significantly reducing the number of out-of-order packets arriving at the receiving end, thereby alleviating the pressure on network interface card (NIC) sorting and reordering and reducing reliance on large-capacity sorting memory. Furthermore, it proposes multiple implementation paths, covering different scenarios from simply modifying network equipment to end-network collaborative transformation, facilitating the selection of appropriate solutions based on actual deployment conditions and requirements.

[0009] In one embodiment of the first aspect, it further includes: Before sending a data packet, the end device sets a target field in the standard protocol header of the data packet. The target field is implemented by modifying reserved fields or adding new fields and is used to identify consecutive data packets in the same data stream.

[0010] In one embodiment of the first aspect, the target field includes: Fill the reserved fields in the network layer core protocol options field or the remote direct memory access extended transport header with preset bit values ​​so that the reserved field values ​​are the same for multiple consecutive data packets; or Insert custom fields before and after the MAC frame header, IP header, or User Datagram Protocol header of the data packet, and fill them with preset bit values ​​so that the custom field values ​​are the same for multiple consecutive data packets.

[0011] In one embodiment of the first aspect, it further includes: The endpoint device establishes an association between the User Datagram Protocol (UDP) source port number and the Remote Direct Memory Access queue pair; Based on the aforementioned association, the User Datagram Protocol (UDP) source port number is set for the data packets corresponding to the remote direct memory access queue. When the network-side device performs hash calculations, the source port number in the five-tuple information is the user datagram protocol source port number that has been set.

[0012] In one embodiment of the first aspect, the association includes: A logical AND operation is performed between the User Datagram Protocol (UDP) source port number and the mask to obtain a new UDP source port number. The same mask value is then assigned to multiple consecutive data packets to make the UDP source port numbers of the multiple consecutive data packets the same.

[0013] In one embodiment of the first aspect, the association further includes: Assign multiple User Datagram Protocol (UDP) source port numbers to a single remote direct memory access queue pair; Multiple consecutive data packets are divided into segments, and one of the multiple User Datagram Protocol (UDP) source port numbers is assigned to each data packet within a segment using a random or round-robin method.

[0014] In a second aspect, this application provides a novel per-packet load balancing device, comprising: The data receiving module is used by network-side devices to receive data packets; The identification module is used to obtain the target field and five-tuple information in the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the five-tuple information includes the source IP address, destination IP address, source port number, destination port number, and transport layer protocol; The hash calculation module is used to perform hash calculations based on the target field and the quintuple information to obtain the hash result; The transmission module is used to determine the outgoing port of the data packet based on the hash result, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the novel per-packet load balancing method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the novel per-packet load balancing method as described above.

[0017] This invention provides a novel per-packet load balancing method, apparatus, device, and storage medium. By employing a more granular segment-based load balancing approach than flow-based load balancing, it breaks hash polarization, enabling traffic to be distributed more evenly across multiple network paths, thus improving network bandwidth utilization and overall performance. By bundling consecutive data packets together for transmission, it significantly reduces the number of out-of-order packets arriving at the receiving end, thereby alleviating the pressure on network interface card (NIC) sorting and reordering and reducing reliance on large-capacity sorting memory. Furthermore, it proposes multiple implementation paths, covering different scenarios from simply modifying network equipment to end-network collaborative transformation, facilitating the selection of appropriate solutions based on actual deployment conditions and needs, achieving end-network collaboration and field innovation, solving the problems of uneven load and congestion in existing technologies, and is applicable to various storage scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the novel per-packet load balancing method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the structure of the RDMA transport layer Base Transport Header provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the IP header format provided by the present invention.

[0022] Figure 4 This is a schematic diagram showing that the option field of the IP data packet provided by this invention is 32 bits.

[0023] Figure 5 This is a schematic diagram of the variable-length extended transmission header of the RDMA transmission layer provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the module connection of the novel per-packet load balancing device provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0026] Figure label: 110: Data receiving module; 120: Identification module; 130: Hash calculation module; 140: Transmission module; 710: Processor; 720: Communication interface; 730: Memory; 740: Communication bus. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the accompanying drawings, not the entire structure.

[0028] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] Before introducing the embodiments of this application, the relevant terms involved in the embodiments of this application are defined as follows: QP (Queue Pair) is a core concept in the RDMA (Remote Direct Memory Access) protocol, referring to a logical communication endpoint consisting of a pair of queues (SQ for sending and RQ for receiving). Each QP independently manages the sending and receiving of data packets without operating system intervention. In end-side modification schemes, one RDMA QP can be associated with multiple UDP source port numbers. By dynamically allocating port numbers, a data stream can be divided into multiple segments (e.g., each segment consists of M packets), achieving fine-grained load balancing of traffic.

[0032] GLB (Global Load Balancing) is a load balancing technology that spans multiple data centers or wide area networks. It improves global resource utilization by dynamically scheduling traffic to the optimal path or node. Packet-by-packet load balancing can be seen as a micro-implementation of GLB in data center networks, avoiding local congestion through path optimization. This invention focuses on multi-path load balancing within data centers (such as ECMP), while GLB is more macro-level; however, the combination of both can solve the problem of cross-node traffic scheduling in intelligent computing networks.

[0033] A flow refers to a sequence of data packets with the same five-tuple (source IP, destination IP, source port, destination port, and transport layer protocol), and is the basic unit of traditional load balancing. Flow-by-flow load balancing (such as Static Load Balance) causes path congestion due to hash polarization, and is particularly unsuitable for large-volume (continuous) flows in AI services. By dividing a flow into multiple segments according to field characteristics (such as PSN prefix), segment-level load balancing is achieved, breaking the five-tuple limitation.

[0034] RDMA (Remote Direct Memory Access) allows a computer to directly access the memory of another computer without CPU involvement, significantly reducing latency and overhead. It optimizes load balancing by utilizing RDMA protocol fields (such as PSN Sequence ID): for example, extracting the first N bits of the PSN for hashing reduces end-side out-of-order reordering pressure. The high bandwidth requirements of RDMA expose the shortcomings of flow-by-flow load balancing, while this invention adapts its low-latency characteristics through field-level hashing.

[0035] ECMP (Equal-Cost Multi-Path) distributes traffic evenly across multiple equal-cost paths to improve bandwidth and reliability. This invention uses ECMP as its basic framework but changes its hash input. Traditional ECMP relies solely on the 5-tuple, while this invention incorporates packet fields (such as PSN and Identification) into the hash parameters, improving the rationality of path selection. It retains ECMP's multi-path capabilities while avoiding hash polarization through field expansion, making it suitable for high-speed network scenarios.

[0036] RoCE (RDMA over Converged Ethernet) is a network protocol that allows RDMA to be implemented over Ethernet. It is divided into RoCEv1 (based on the Ethernet link layer) and RoCEv2 (based on the UDP / IP layer). This invention optimizes load balancing for the RoCEv2 protocol by utilizing the PSN field. RoCEv2 packets contain a 24-bit PSN sequence number, and the first N bits are extracted as hash input. The UDP port is modified by dynamically changing the UDP source port number (such as through masking) to disguise multiple flows and balance the path. The high throughput requirements of RoCEv2 conflict with the out-of-order pressure of packets. This invention controls the out-of-order rate while ensuring bandwidth through segment-level hashing.

[0037] The following is combined Figure 1 A novel packet-by-packet load balancing method according to the present invention includes: Step 100: The network-side device receives the data packet; Step 200: Obtain the target field and 5-tuple information from the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the 5-tuple information includes the source IP address, destination IP address, source port number, destination port number, and transport layer protocol; Step 300: Perform a hash calculation based on the target field and the quintuple information to obtain the hash result; Step 400: Based on the hash result, determine the outgoing port of the data packet, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0038] This invention proposes a novel per-packet load balancing method for intelligent computing scenarios. One approach utilizes a field in a standard protocol that identifies several consecutive data packets within a flow as hash input, ensuring that the hash results of these consecutive data packets within a flow are identical, thus reducing the reordering pressure on the network interface card (NIC). Another approach involves adding a special field to the packet header of existing standard protocols. For several consecutive data packets within a flow, this special field is filled with the same value, and this value is used as the ECMP hash input, achieving the same effect.

[0039] The per-packet load balancing method of this invention is similar to the flowlet technology of DLB. The DLB approach involves finding a large time slot between consecutive data packets in a flow, dividing the flow into multiple sub-flows called flowlets, and performing flowlet-level load balancing. However, due to the bursty nature of data flows in intelligent computing scenarios, these time slots between data packets are not as pronounced as in TCP. Therefore, a mechanism is needed to achieve load balancing at the level of large data packet segments to reduce the effect of out-of-order reordering on the receiving network interface card (NIC). Currently, for intelligent computing scenarios, there are two approaches in terms of routing protocols: 1. Utilizing fields in standard routing protocols that can identify data packets, hashing a continuous segment of data packets as a whole according to a fixed hash rule; 2. Adding a special field to the standard routing protocol, hashing the value of this field, and combining this with modifications to the endpoint (server NIC) and network side (switch).

[0040] Specifically, if the target field is the sequence number of a remote direct memory access packet on converged Ethernet, then a preset number of bits is taken from the sequence number to participate in the hash calculation. If the target field is the header identifier field of the network layer core protocol data packet, then a preset number of bits are taken from the header identifier field to participate in the hash calculation.

[0041] Before sending a data packet, the end device sets a target field in the standard protocol header of the data packet. The target field is implemented by modifying reserved fields or adding new fields and is used to identify consecutive data packets in the same data stream.

[0042] The target fields include: Fill the reserved fields in the network layer core protocol options field or the remote direct memory access extended transport header with preset bit values ​​so that the reserved field values ​​are the same for multiple consecutive data packets; or Insert custom fields before and after the MAC frame header, IP header, or User Datagram Protocol header of the data packet, and fill them with preset bit values ​​so that the custom field values ​​are the same for multiple consecutive data packets.

[0043] In one specific embodiment, no modification is made to the end side, but the network side is modified: The core approach utilizes the high-order bits of the existing protocol sequence field; it hashes the PSN Sequence ID (the first N bits out of 24 bits) of the RoCE packet with a 5-tuple to calculate the packet's outgoing port; it also hashes the Identification header (the first N bits out of 16 bits) of the IP packet with a 5-tuple to calculate the packet's outgoing interface. The key is leveraging the existing monotonically increasing sequence number field in the packet header. The sending end transmits packets normally, requiring no modifications whatsoever. All modifications are concentrated on the network switch.

[0044] Different fields are selected for different protocols. For RoCEv2 traffic: the PSN field in the RDMA transport layer header is used. The PSN is 24 bits long and is used to identify the order of data packets; it increments by 1 with each packet sent. For ordinary IP traffic: the Identification field in the IP header is used. This field is 16 bits long and typically also increments with each data packet.

[0045] Hash Algorithm Modification: The ECMP / WCMP hash algorithm of the switch has been modified. The new hash input is no longer just the traditional 5-tuple, but a 5-tuple plus the first N bits of the sequence number field; for example, N=8. For the PSN field, the switch extracts the high 8 bits (bits 24 to 17) of the PSN during hash calculation; since the 5-tuple is fixed in a stream, the variation in the hash result is entirely determined by these first N bits.

[0046] A data segment is formed, and the hash result remains unchanged as long as the high N bits of the PSN or Identification remain constant. The number of packets corresponding to one high N bit value is 2^(total bits - N). For example, when N=8, a "data segment" contains 2^(2^4 - 8) = 65536 consecutive packets. Within these 65536 packets, their high 8 bits are the same, so they will be forwarded to the same path. The data segment will only switch when the PSN increments, causing a change in the high 8 bits; the hash result may change, thus switching to another path.

[0047] Significantly reduces out-of-order delivery, drastically reducing the potential out-of-order range from the "per packet" level to the "per tens of thousands of packets" level (the specific value is determined by N), greatly alleviating the reordering pressure and memory requirements of the receiving network card. Achieves load balancing; since the high N bits of data segments in different flows vary randomly, they are hashed more evenly onto different paths, effectively breaking the hash polarization problem of flow-by-flow load balancing and improving network bandwidth utilization. Deployment advantages: zero requirements on existing servers, operating systems, and network card drivers, achieving "seamless upgrades." Network operators can introduce this optimization into their existing networks simply by upgrading switch software or configurations, resulting in low deployment costs and quick results.

[0048] In another specific embodiment, the terminal side is modified, but the network side is not modified, and the UDP source port number is intelligently manipulated. Regarding the modification of UDP source port numbers in ROCE v2, there are two approaches: a) After associating a 5-tuple UDP source port number with a QP, divide it into segments of M data packets and regularly fill the tail of the UDP source port number with fixed bits, then allocate these segments to the data packets; b) Assign multiple UDP source port numbers to an RDMA QP, divide the data packets sent by a QP into segments of M consecutive data packets, and randomly or in a round-robin fashion assign one of the aforementioned UDP source port numbers to all data packets within each segment. These two methods achieve the goal of dividing the data stream sent by a QP into multiple streams.

[0049] The core idea is for the sending network interface card (NIC) to actively and systematically change the five-tuple characteristics of data packets, thereby "tricking" the standard switch into performing segment-based load balancing. The switch requires no modification; it simply runs the standard ECMP hash algorithm.

[0050] In RoCEv2, the UDP destination port is fixed (4791), but the source port is usually randomly generated by the driver or fixed based on QPs (queue pairs). Using a bitmasking method, the end-side network interface card (NIC) maintains a state that assigns a specific segment identifier to every M consecutive packets of each QP. The NIC uses bitmasking to force the lower bits (e.g., the lower 8 bits) of the UDP source port number to the value of the current segment identifier, while keeping the higher bits unchanged to maintain the association with the QP. For a standard switch, the 5-tuples of every M packets (due to different source ports) appear to come from different micro-streams, thus being hashed to different paths. However, M packets within the same segment, because their source ports are set to the same value, are considered to be from the same micro-stream and follow the same path.

[0051] Based on a multi-port pool approach, a port pool network interface card (NIC) containing multiple UDP source ports is pre-allocated to a QP (Queries Pointer). When sending data, every M consecutive data packets are considered a segment, and a source port from the port pool is randomly or round-robin-based for all packets within the segment. By precisely controlling the data segment size, the segment size M can be precisely controlled by the endpoint NIC (e.g., every 1000 packets per segment), unlike other methods where the endpoint remains unchanged while the network side is modified, and the size is determined by the bit length of the field (e.g., 65536 packets), offering greater flexibility. It is compatible with existing network equipment, requiring no upgrades to any switches in the data center network. As long as the network supports the ECMP standard, it can benefit from the improved load balancing, protecting existing investments. The effectiveness depends on deployment; its advantages depend on the widespread adoption and modification of endpoint NICs. Once deployed, it can effectively resolve congestion and precisely control out-of-order delivery.

[0052] In another specific embodiment, the end-side and network-side are jointly modified. By using reserved fields or newly added custom fields, based on the reserved fields or newly added fields in the packet header of the standard network protocol, several consecutive data packets in a flow are identified. Combined with the five-tuple for hashing, the outgoing interface of the data packet is calculated. This is the most flexible and thorough solution. Through end-network collaboration, a dedicated signaling channel is created for load balancing.

[0053] During field selection and definition, reserved fields are used to retrieve unused reserved fields from existing network protocols (such as IP options, RDMA base transport headers, and extended transport headers). A rule is defined, for example, to use 8 consecutive bits as a load balancing segment identifier. New custom fields are added to insert a new custom header or field into the protocol stack, such as within the VXLAN header, as an IP option, or even between the MAC frame header and the IP header. This new field is dedicated to carrying the segment identifier.

[0054] On the sending end, when constructing data packets, the network interface card (NIC) sets the "segment identifier" field to the same value for every M consecutive data packets; M can be flexibly set by the NIC policy. The next M packets then update this field value to a different value. On the network side, the switch is modified to recognize and parse this specific "segment identifier" field. Its hash algorithm input becomes a 5-tuple + segment identifier. Therefore, packets within the same segment (with the same segment identifier) ​​travel along the same path, while packets from different segments (with different segment identifiers) may travel along different paths.

[0055] By coordinating modifications on both the endpoint and network sides, flexibility and control are maximized. The data segment size M and the identifier bit width N can be defined as needed, achieving the most granular control. Network administrators can adjust policies according to the needs of different applications (such as latency sensitivity). Because it is a mechanism specifically designed for load balancing, it can achieve optimal traffic distribution and the lowest, predictable level of out-of-order traffic. Furthermore, the endpoint-network coordinating design represents the future direction of data center network optimization, laying the foundation for more advanced network functions (such as in-band network telemetry and application-aware networking). This requires both endpoint network interface cards (NICs) and network switches to support new protocol standards or extensions, and is typically suitable for newly built data centers or self-developed systems pursuing ultimate performance.

[0056] Specifically, the value is hashed by combining the RDMA transport layer PSN Sequence ID with a 5-tuple; see reference. Figure 2 This is the structure of the RDMA transport layer Base Transport Header, where the PSN field occupies 3 bytes (24 bits). N is set, where N is greater than 1 and less than 24. Values ​​are taken starting from the leftmost position of the PSN ID, up to the Nth position. The first N bits of the PSN are extracted as one of the hash parameters, combined with the 5-tuple to form a 6-tuple for hashing. Since the 5-tuple for a flow is fixed, the variable is the first N bits of the PSN field. The effect is that data packets with the same first N bits of the PSN in a flow (2^(24-N) consecutive data packets) will have the same hash result and will therefore be forwarded to the same path, thus reducing the number of out-of-order packets on the receiving network interface card.

[0057] The hash value is obtained by combining the IP header Identification with the 5-tuple, as shown in the reference. Figure 3 The IP header format includes a 2-byte (16-bit) Identification field. This value increments by 1 for each packet sent by the host. A value N is set, where N is greater than 1 and less than 16. Values ​​are taken from the first N bits of the Identification ID, starting from the leftmost bit. The first N bits of the Identification ID are extracted as hash parameters and combined with a 5-tuple to form a 6-tuple for hashing. Since the 5-tuple for a flow is fixed, the variable is the first N bits of the Identification field. The desired effect is that, ignoring the specific value changes from the last N+1 bits to the 16th bit of the 16-bit Identification field, packets with the same first N bits (2^(24-N) consecutive packets) will have the same hash result and will therefore be forwarded to the same path. This reduces the complexity of packet-by-packet hashing and reduces the number of out-of-order packets on the receiving network interface card.

[0058] The first scenario involves hashing the UDP source port number associated with a QP. In the RoCE v2 protocol, RDMA data is transmitted via UDP. According to the RoCE v2 protocol specification, the UDP destination port is fixed (des port = 4791), but the UDP source port is determined by the RDMA driver using hash calculation. Typically, a 5-tuple is associated with one RDMA QP. Our approach is to first associate the UDP source port field with a special mask 1 of the same length and perform an operation (&&) on the resulting 5-tuple to create a QP. Then, the original source port is replaced with the new source port obtained by the operation (&&) on the special mask 2, forming a new 5-tuple. A special mask 1 and 2 are assigned to a fixed number of M packets. The network-side switch then hashes each data packet segment based on the new 5-tuple. The specific process is as follows: Set N, where N is greater than 0 and less than 16. The network interface card (NIC) on the end side starts taking values ​​from the leftmost bit of the UDP source port number, up to the Nth bit. The source port is then ANDed (&&) with a mask of the same length (bits from the leftmost bit to the Nth bit are 1, and bits from N+1 to 16 are all 0). The resulting source port is a 5-tuple corresponding to one RDMA QP. Special mask 1 is defined as: bits from the leftmost bit to the Nth bit are 1, and bits from N+1 to 16 are all 0. After the new 5-tuple corresponds to one RDMA QP, the new source port is ANDed (&&) with a mask of the same length (bits from the leftmost bit to the Nth bit are 0, and bits from N+1 to 16 range from all 0 to all 1, with the length of the all-0 and all-1 bits being 17 minus N). The resulting source port is the new 5-tuple. Special mask 2 is defined as follows: the first N bits from the left are 1, and the bits from N+1 to 16 increment from all 0s to all 1s. The length of the bits between all 0s and all 1s is 17 minus N. During this process, M consecutive data packets are assigned one special mask 1 and one special mask 2.

[0059] The second scenario involves assigning N UDP source port numbers to an RDMA QP and maintaining a mapping table between the N UDP source port numbers and the QP. Data packets sent by a QP are divided into segments of M consecutive packets, and each segment is randomly or in a round-robin fashion assigned one of the aforementioned N UDP source port numbers. This method effectively splits the data stream sent by a QP into multiple streams. The effect of these two methods is that after the end-side network interface card (NIC) is modified as described above, the network-side switch changes from using random source port parameters as a five-tuple hash for each stream to using a controllable and regular source port parameter five-tuple hashed in units of M consecutive data packets. This achieves balanced load balancing across all exits and effectively controls the number of out-of-order packets on the receiving NIC.

[0060] In standard network protocols, special fields are added to the packet header and hashed using a 5-tuple. The current RDMA V2 intelligent computing protocol adds special fields in two ways: 1. Filling in the reserved fields in the current network protocol with values; 2. Inserting custom fields into any position in the existing packet header.

[0061] Regarding the first scenario, the reserved fields in current network protocols, specifically the option fields in IP packets, are referenced. Figure 4 32 bits; Reference Figure 2 This is the structure of the RDMA transport layer Base Transport Header, where reserved fields occupy 7 bits; see reference. Figure 5 For RDMA transport layer, the Extended Transport Header (ETH) of variable length is reserved for UD and RD type services, with 8 bits reserved for RC type services.

[0062] For the reserved fields already present in the aforementioned network protocol, set N, where N is greater than 0 and less than the maximum number of bits in the reserved field. The bits from the left to the Nth bit range from all 0s to all 1s, with the length of the all-0s and all-1s bits being N. The server-side network interface card (NIC) allocates one reserved field value for every M data packets. The switch extracts the N-bit value of this reserved field as a hash parameter and combines it with a 5-tuple to form a 6-tuple for hashing.

[0063] Insert a custom field at any position in the header of the existing data packet. The network card on the server side inserts custom fields before and after the MAC frame header, IP packet header, and UDP header of the network packet. The size of the field is N bits. Starting from the left, the N-bit value starts from all 0s and goes from binary to all 1s. The bit lengths of all 0s and all 1s are N. The network card on the server side allocates a custom field value according to the order size of M data packets. The switch extracts the N-bit value of this custom field as a hash parameter and combines it with the five-tuple to form a six-tuple for hashing.

[0064] In the present invention, a method of taking values and hashing by combining the RDMA transport layer PSN Sequence ID with the five-tuple: Starting from the leftmost front of the PSN ID, take values until N (0 < N < 24) bits. Extract the first N-bit value of the PSN and combine it with the five-tuple to form a six-tuple for hashing. Since the five-tuple of a flow is fixed, the changing quantity is the first N-bit value of the PSN field. The achievable effect is that the hash results of data packets with the same first N bits of the PSN in a flow (2^(24-N) consecutive data packets) are the same and thus are forwarded to the same path. Ignore the specific numerical changes of the last N + 1 to 24 bits, and achieve that the changes in the first N bits represent a continuous data packet for hashing. A method of taking values and hashing by combining the Identification in the IP header with the five-tuple: Starting from the leftmost front of the Identification ID, take values until N (0 < N < 16) bits. Extract the first N-bit value of the Identification ID and combine it with the five-tuple to form a six-tuple for hashing. Since the five-tuple of a flow is fixed, the changing quantity is the first N-bit value of the Identification field. The achievable effect is that the hash results of data packets with the same first N bits of the Identification in a flow (2^(24-N) consecutive data packets) are the same and thus are forwarded to the same path. Ignore the specific numerical changes of the last N + 1 to 16 bits, and achieve that the changes in the first N bits represent a continuous data packet for hashing.

[0065] For ROCE v2 UDP source port number modification, the end-side network interface card (NIC) has two approaches: 1. First, associate a QP with the 5-tuple resulting from the UDP Sourceport field and a special mask of the same length (&&). Then, replace the original Sourceport with the new Sourceport resulting from the special mask (&&) and the new Sourceport. Assign a special mask (1) and 2 for every M consecutive packets. The network-side switch performs packet hashing segment by segment based on the new 5-tuple. The definitions of special masks 1 and 2 are described in Chapter 5, Section 3. 2. Assign multiple UDP source port numbers to an RDMA QP. Divide the data packets sent by a QP into segments of M consecutive packets, and assign one of the aforementioned UDP source port numbers to all packets within each segment randomly or in a round-robin fashion. These two methods achieve the goal of splitting the data stream sent by a QP into multiple streams.

[0066] For cases where reserved fields already exist in the network protocol, N-bit values ​​are systematically filled into the reserved fields. The server network card fills the same reserved field into M packets, and the switch performs hashing based on the reserved field value combined with the 5-tuple. For cases where custom fields are inserted before and after the packet header in the network protocol, N-bit values ​​are systematically filled into the custom fields. The server network card fills the same custom field into M packets, and the switch performs hashing based on the custom field value combined with the 5-tuple.

[0067] By extracting existing standard fields from standard routing protocols and utilizing reserved fields or newly added fields before and after data packets in standard network protocols, combined with the modifications to end-side network cards and network-side switches, a hashing method is achieved where a segment of continuous data packets in a flow is a single feature. This hashing method effectively reduces the ability of the receiving server's network card to reorder data packets and avoids the problem of uneven load balancing in intelligent computing scenarios.

[0068] refer to Figure 6 The present invention also discloses a novel packet-by-packet load balancing device, comprising: The data receiving module 110 is used by network-side devices to receive data packets; The identification module 120 is used to obtain the target field and five-tuple information in the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the five-tuple information includes the source IP address, destination IP address, source port number, destination port number, and transport layer protocol; Hash calculation module 130 is used to perform hash calculation based on the target field and the quintuple information to obtain a hash result; The transmission module 140 is used to determine the outgoing port of the data packet based on the hash result, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0069] Wherein, if the target field is the sequence number sequence identifier of the remote direct memory access packet on converged Ethernet, then a preset number of bits are taken from the sequence number sequence identifier to participate in the hash calculation; If the target field is the header identifier field of the network layer core protocol data packet, then a preset number of bits are taken from the header identifier field to participate in the hash calculation.

[0070] Before sending a data packet, the end device sets a target field in the standard protocol header of the data packet. The target field is implemented by modifying reserved fields or adding new fields and is used to identify consecutive data packets in the same data stream.

[0071] The target fields include: Fill the reserved fields in the network layer core protocol options field or the remote direct memory access extended transport header with preset bit values ​​so that the reserved field values ​​are the same for multiple consecutive data packets; or Insert custom fields before and after the MAC frame header, IP header, or User Datagram Protocol header of the data packet, and fill them with preset bit values ​​so that the custom field values ​​are the same for multiple consecutive data packets.

[0072] The endpoint device establishes an association between the User Datagram Protocol (UDP) source port number and the Remote Direct Memory Access queue pair; Based on the aforementioned association, the User Datagram Protocol (UDP) source port number is set for the data packets corresponding to the remote direct memory access queue. When the network-side device performs hash calculations, the source port number in the five-tuple information is the user datagram protocol source port number that has been set.

[0073] The relationships include: A logical AND operation is performed between the User Datagram Protocol (UDP) source port number and the mask to obtain a new UDP source port number. The same mask value is then assigned to multiple consecutive data packets to make the UDP source port numbers of the multiple consecutive data packets the same.

[0074] Assign multiple User Datagram Protocol (UDP) source port numbers to a single remote direct memory access queue pair; Multiple consecutive data packets are divided into segments, and one of the multiple User Datagram Protocol (UDP) source port numbers is assigned to each data packet within a segment using a random or round-robin method.

[0075] This invention provides a novel per-packet load balancing device that breaks hash polarization by providing finer-grained segment-by-segment load balancing than flow-based load balancing, enabling traffic to be distributed more evenly across multiple network paths, thus improving network bandwidth utilization and overall performance. By bundling consecutive data packets together for transmission, it significantly reduces the number of out-of-order packets arriving at the receiving end, thereby alleviating the pressure on network card sorting and reordering and reducing reliance on large-capacity sorting memory. Furthermore, it proposes multiple implementation paths, covering different scenarios from simply modifying network equipment to end-to-end network collaborative transformation, facilitating the selection of appropriate solutions based on actual deployment conditions and needs, achieving end-to-end network collaboration and field innovation, solving the problems of uneven load and congestion in existing technologies, and is applicable to various storage scenarios.

[0076] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a novel per-packet load balancing method. This method includes: the network-side device receiving data packets; obtaining a target field and a five-tuple information from the data packets, wherein the target field identifies the data stream segment to which the data packet belongs, and the five-tuple information includes a source IP address, a destination IP address, a source port number, a destination port number, and a transport layer protocol; performing a hash calculation based on the target field and the five-tuple information to obtain a hash result; and determining the outgoing port of the data packet based on the hash result, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0077] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the novel per-packet load balancing method provided by the above methods. The method includes: a network-side device receiving a data packet; obtaining a target field and a 5-tuple information from the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the 5-tuple information includes a source IP address, a destination IP address, a source port number, a destination port number, and a transport layer protocol; performing a hash calculation based on the target field and the 5-tuple information to obtain a hash result; and determining the outgoing port of the data packet based on the hash result, so that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0079] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a novel per-packet load balancing method provided by the methods described above. The method includes: a network-side device receiving a data packet; obtaining a target field and a 5-tuple information from the data packet, wherein the target field is used to identify the data stream segment to which the data packet belongs, and the 5-tuple information includes a source IP address, a destination IP address, a source port number, a destination port number, and a transport layer protocol; performing a hash calculation based on the target field and the 5-tuple information to obtain a hash result; and determining the outgoing port of the data packet based on the hash result, such that data packets within the same data stream segment are transmitted through the path corresponding to the same outgoing port.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A novel per-packet load balancing method characterized in that, The method comprises the following steps: The network-side device receives a data packet; Obtain a target field and a five-tuple information in the data packet, wherein the target field is used to identify a data flow segment to which the data packet belongs, and the five-tuple information comprises a source IP address, a destination IP address, a source port number, a destination port number and a transport layer protocol; Hash calculation is performed according to the target field and the five-tuple information to obtain a hash result; According to the hash result, the out port of the data packet is determined, so that the data packets in the same data flow segment are transmitted through the path corresponding to the same out port.

2. The novel per-packet load balancing method according to claim 1, wherein, The target field is a sequence number sequence identifier of a remote direct memory access data packet on a converged Ethernet, and a preset number of bits in the sequence number sequence identifier is used to participate in the hash calculation; The target field is a header identifier field of a network layer core protocol data packet, and a preset number of bits in the header identifier field is used to participate in the hash calculation.

3. The novel per-packet load balancing method according to claim 1, wherein, The method further comprises the following steps: Before sending a data packet, the end-side device sets a target field in a standard protocol packet header of the data packet, wherein the target field is implemented by modifying a reserved field or adding a new field, and is used to identify continuous data packets in the same data flow.

4. The novel per-packet load balancing method according to claim 3, wherein, The target field comprises the following steps: A reserved field in a network layer core protocol option field or a remote direct memory access extended transmission header is filled with a preset number of values, so that the reserved field values of continuous multiple data packets are the same; or A self-defined field is inserted before or after a MAC frame header, an IP header or a user datagram protocol header of a data packet header, and is filled with a preset number of values, so that the self-defined field values of continuous multiple data packets are the same.

5. The novel per-packet load balancing method according to claim 1, wherein, The method further comprises the following steps: The end-side device establishes an association relationship between a user datagram protocol source port number and a remote direct memory access queue pair; Based on the association relationship, a user datagram protocol source port number is set for the data packets corresponding to the remote direct memory access queue pair; When the network-side device performs hash calculation, the source port number in the five-tuple information is the set user datagram protocol source port number.

6. The novel per-packet load balancing method according to claim 5, wherein, The association relationship comprises the following steps: The user datagram protocol source port number is logically ANDed with a mask to obtain a new user datagram protocol source port number, and the same mask value is allocated to continuous multiple data packets, so that the user datagram protocol source port numbers of the continuous multiple data packets are the same.

7. The novel per-packet load balancing method according to claim 5, wherein, The association relationship further comprises the following steps: A plurality of user datagram protocol source port numbers are allocated to one remote direct memory access queue pair; Continuous multiple data packets are divided into segments, and one of the plurality of user datagram protocol source port numbers is allocated to each data packet in the segment in a random or polling manner.

8. A novel per-packet load balancing device, characterized by, The method comprises the following steps: A data receiving module is configured to receive a data packet by a network-side device; An identification module is configured to obtain a target field and a five-tuple information in the data packet, wherein the target field is used to identify a data flow segment to which the data packet belongs, and the five-tuple information comprises a source IP address, a destination IP address, a source port number, a destination port number and a transport layer protocol; A hash calculation module is configured to perform hash calculation according to the target field and the five-tuple information to obtain a hash result; The transmission module is configured to determine an out port of the data packet according to the hash result, so that data packets in the same data flow segment are transmitted through a path corresponding to the same out port.

9. An electronic device, comprising: The electronic device comprises a processor and a memory for storing a computer program capable of running on the processor; wherein the processor is configured to run the computer program to perform the novel per-packet load balancing method of any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the novel per-packet load balancing method of any one of claims 1 to 7.

Images