A high-performance software RoCE data transmission method, electronic equipment and storage medium

Abstract

This invention discloses a high-performance software RoCE data transmission method, electronic device, and storage medium. The method includes: configuring a policy that allows disabling invariant cyclic redundancy check (CRC), and negotiating the disabling of CRC based on the policy when establishing a RoCE connection with a peer node; storing the data to be transmitted in a first memory region, generating a protocol header corresponding to the data to be transmitted, and storing the protocol header in a second memory region; constructing a distributed aggregation list based on the first and second memory regions; and reading the data to be transmitted and the protocol header from the first and second memory regions respectively according to the distributed aggregation list, assembling them into a RoCE message, and sending it to the peer node. In this invention, disabling CRC can improve RoCE throughput and reduce latency; constructing a distributed aggregation list can avoid frequent CPU participation in memory copying, allowing the network card hardware to automatically assemble and send the message, thus reducing cache misses.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a high-performance software RoCE data transmission method, electronic device, and storage medium. Background Technology

[0002] Remote Direct Memory Access (RDMA) is a high-performance network communication technology that enables local devices to directly access the memory of remote hosts. The data exchange process bypasses the operating system kernel and CPU processing, reducing communication latency to the microsecond level and effectively improving overall system data throughput. This technology is widely used in scenarios with extremely high network performance requirements, such as AI training, distributed storage, and high-performance computing.

[0003] Currently, the industry mainly adopts two technical approaches for RDMA deployment: InfiniBand (IB) and RDMA over Converged Ethernet (RoCE). InfiniBand (IB) is a dedicated high-speed networking solution tailored for RDMA, offering superior transmission performance. However, it requires dedicated hardware for network cards and switches, resulting in higher deployment costs and incompatibility with existing Ethernet devices. RoCE, on the other hand, is based on the existing RDMA over Converged Ethernet protocol, encapsulating the InfiniBand protocol into IP and UDP packets for transmission. Its performance indicators are close to those of InfiniBand, and it can reuse existing Ethernet hardware in data centers, resulting in lower deployment costs. As a result, RoCE is gradually becoming the mainstream choice for RDMA deployment in data centers.

[0004] Current RoCE implementations fall into two categories: native hardware network interface card (NIC) implementations and software-based RDMA (Linux Soft RDMA over Ethernet, RXE) emulation. Large-scale data centers, constrained by construction budgets, cannot replace all NICs across all nodes at once. Existing ordinary NICs cannot directly connect to hardware-based RoCE nodes, easily creating isolated network silos. To overcome hardware compatibility limitations, the industry commonly uses RXE software RoCE to emulate RoCE communication on general-purpose NICs; however, this software solution has significant performance limitations.

[0005] The RoCE protocol mandates that every packet undergo an Invariant Cyclic Redundancy Check (ICRC) operation. This operation cannot be accelerated using the CPU's built-in hardware CRC instruction; it relies entirely on serial software computation, continuously consuming significant CPU resources. Furthermore, the software solution relies on the CPU to assemble Ethernet, Internet Protocol (IP), and User Datagram Protocol (UDP) headers packet by packet. This, coupled with frequent system context switching triggered by network card packet reception interrupts, and the multiple memory copies between user and kernel modes, along with frequent cache misses, further exacerbates system resource consumption. Real-world testing shows that in large packet scenarios, the single-core throughput of RXE can only reach a maximum of 10Gbps, while in small packet transmission scenarios, throughput drops back to 1Gbps, with transmission latency ranging from 10μs to 40μs. Overall, its performance is an order of magnitude lower than that of native hardware RoCE.

[0006] In summary, existing software RoCE solutions are constrained by multiple factors such as ICRC calculation and memory copying, resulting in high resource consumption and poor transmission performance. The industry urgently needs a high-performance software RoCE data transmission solution that is compatible with general-purpose network cards and balances compatibility and transmission efficiency. Summary of the Invention

[0007] This invention provides a high-performance software RoCE data transmission method, electronic device, and storage medium that does not rely on dedicated hardware, has performance close to hardware RoCE, strong compatibility, and is suitable for ordinary network cards and virtualization environments.

[0008] According to one aspect of the present invention, a high-performance software RoCE data transmission method is provided, comprising: Configure a policy that allows disabling Invariant Cyclic Redundancy Check (CRC), and negotiate to disable CRC based on the policy when establishing a RoCE connection with the peer node. The data to be transmitted is stored in the first memory area, a protocol header corresponding to the data to be transmitted is generated, and the protocol header is stored in the second memory area. Construct a distributed aggregation list based on the first memory region and the second memory region; According to the distributed aggregation list, the data to be transmitted and the protocol message header are read from the first memory region and the second memory region respectively, to assemble them into a RoCE message and send it to the peer node.

[0009] According to another aspect of the present invention, a high-performance software RoCE data transmission device is provided, comprising: The policy configuration module is used to configure a policy that allows the invariant cyclic redundancy check to be disabled, and to negotiate the disabling of the invariant cyclic redundancy check based on the policy when establishing a RoCE connection with the peer node. The data caching module is used to store the data to be transmitted into a first memory area, generate a protocol header corresponding to the data to be transmitted, and store the protocol header into a second memory area. The list building module is used to build a scatter-aggregated list based on the first memory region and the second memory region; The message transceiver module is used to read the data to be transmitted and the protocol message header from the first memory area and the second memory area respectively according to the distributed aggregation list, so as to assemble them into a RoCE message and send it to the peer node.

[0010] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the high-performance software RoCE data transmission method according to any embodiment of the present invention.

[0011] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the high-performance software RoCE data transmission method according to any embodiment of the present invention.

[0012] Traditional RoCE communication performs real-time calculation and verification of Invariant Cyclic Redundancy Check (CRC) by default throughout the entire process, which continuously consumes CPU computing power. The technical solution of this invention configures a policy that allows disabling CRC and negotiates to disable this check function when establishing a link with the peer node, reusing the original check functions of Ethernet, IP, and UDP to achieve message integrity judgment, effectively reducing CPU usage and improving RoCE message throughput performance. Traditional message encapsulation requires the CPU to copy discrete service data and protocol headers to contiguous memory before sending them to the network card. The technical solution of this invention stores the data to be transmitted and the protocol headers in two independent memory locations, constructs a distributed aggregation list, and then uses hardware to read data across regions and assemble messages according to the list, eliminating the CPU memory copying operation and optimizing message sending efficiency.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating a high-performance software RoCE data transmission method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the RoCE message layered encapsulation structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the location of various verification fields in a RoCE message provided in an embodiment of the present invention; Figure 4 A flowchart illustrating another high-performance software RoCE data transmission method provided in this embodiment of the invention; Figure 5 A schematic diagram of the principle of assembling messages using discrete memory with the aid of scatter aggregation descriptors provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the structure of a high-performance software RoCE data transmission device provided in an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an electronic device for implementing the high-performance software RoCE data transmission method of this invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Figure 1 This is a flowchart illustrating a high-performance software RoCE data transmission method provided in an embodiment of the present invention. This embodiment is applicable to situations where RoCE protocol stacks are implemented in software to improve data transmission efficiency. This method can be executed by a high-performance software RoCE data transmission device, which can be implemented in hardware and / or software and can be configured in a computer device. Figure 1 As shown, the method specifically includes the following steps: S110. Configure a policy that allows disabling the invariant cyclic redundancy check (CRRC), and negotiate to disable the CRRC based on the policy when establishing a RoCE connection with the peer node.

[0019] Invariant Cyclic Redundancy Check (CCRC) can be understood as a verification mechanism in the RoCE protocol used to detect whether errors occur in data packets during transmission. It should be noted that in traditional schemes, the fields corresponding to CCRC require real-time calculation and filling by the sending end and real-time verification by the receiving end.

[0020] A peer node refers to a communication peer device in a network link that establishes a RoCE link with the local device. For example, both the local device and the peer node can be servers, and they can remotely access each other's memory based on the RoCE link.

[0021] Specifically, the local device can pre-configure the policy for enabling and disabling ICRC. Subsequently, during the RoCE link establishment process, the local device negotiates with the peer node based on the configured policy to determine whether to disable the ICRC function. If the negotiation is successful, both the local device and the peer node disable the ICRC function. This flexible disabling of verification operations through link negotiation reduces the CPU overhead during message transmission and reception.

[0022] In some optional implementations, the configuration allows for a strategy to disable invariant cyclic redundancy check (CRC), including setting a configuration item that allows disabling CRC via kernel module parameters or user-space configuration files.

[0023] Kernel module parameters can be configuration parameters entered when the operating system kernel loads the driver. User-space configuration files refer to configuration documents that run at the user application layer and whose parameters can be customized and modified. Configuration items can be understood as parameter entries used to mark the enable / disable status of switches.

[0024] Specifically, operators can choose to modify the kernel module parameters corresponding to the network card driver, or they can edit the user-space configuration file. By using either of these two configuration methods, they can write configuration items for enabling and disabling permissions, thereby completing the policy implementation.

[0025] In one possible implementation, negotiating the closure of the Invariant Cyclic Redundancy Check (CRC) based on the policy when establishing a RoCE connection with the peer node includes: sending a connection request carrying a closure support flag to the peer node; determining, based on the connection request, whether the peer node supports closing the CRC; and, if the peer node supports closing the CRC, disabling the real-time calculation and real-time verification of the CRC.

[0026] The disable support flag can be a binary flag field carried in the chain establishment message. This field is used to inform the peer node that the local device has the ability to disable ICRC.

[0027] Specifically, when a local device initiates RoCE chain establishment, it can include a disable support flag in the connection request message during the handshake and send it to the peer node. After waiting for a response message from the peer node, the local device can determine whether the peer node supports disabling ICRC based on the response content. After confirming that the peer node supports disabling ICRC, the local device disables real-time ICRC calculation and verification operations.

[0028] For example, ICRC can be turned off through the following procedure: 1. Pre-configuration phase: The "Allow ICRC to be disabled" policy can be configured on the local server via kernel module parameters or user-space configuration files.

[0029] 2. Connection Negotiation Phase: When two servers establish a RoCE connection, the initiator includes the "Support disabling ICRC" flag in the connection request; If the recipient also supports this feature, a confirmation response will be returned; After reaching an agreement, both parties will simultaneously disable the ICRC filling and verification functions.

[0030] In this embodiment of the invention, the method further includes: when the invariant cyclic redundancy check is disabled, the field corresponding to the invariant cyclic redundancy check is no longer calculated when sending a message; when receiving a message, the integrity of the message is determined based on Ethernet frame check, IP check, and UDP check.

[0031] Ethernet frame checksums can be the Frame Check Sequence (FCS) mechanism built into the Ethernet link layer, such as the Ethernet Frame Check Sequence (ETHFCS). IP checksums refer to the header checksum mechanism built into the IP protocol header, such as the IPv4 Header Checksum (IPv4 checksum). UDP checksums can be understood as the data verification mechanism configured in the UDP transport layer, such as the UDP checksum (UDP checksum). Message integrity refers to the absence of packet loss, errors, or other abnormalities during message transmission.

[0032] Specifically, after negotiating the disabling of ICRC, the local device can act as a sender to send data to the peer node or as a receiver to receive data from the peer node. In this case, when the local device acts as a sender, it can skip the ICRC field calculation step; when the local device acts as a receiver, it no longer verifies the ICRC field but can instead complete the message integrity determination based on the built-in verification mechanisms of the link layer, network layer, and transport layer. Optionally, a 4-byte ICRC field can be retained to enhance protocol compatibility.

[0033] It should be noted that the RoCE protocol requires ICRC verification for every packet by default, and this algorithm cannot be accelerated using CPU hardware CRC instructions, relying entirely on software calculation and consuming significant CPU resources. Therefore, this embodiment of the invention utilizes the multi-layered verification mechanisms already present in ordinary network cards, such as ETH FCS, IPv4 checksum, and UDP checksum, to eliminate the real-time calculation and verification of ICRC in the RoCE protocol while ensuring packet integrity detection capabilities, thus significantly reducing CPU overhead.

[0034] Specifically, RoCE messages are carried on top of the UDP / IP / Ethernet protocol stack, and these underlying protocols all have data integrity verification capabilities: ETH FCS: Hop-by-hop verification, detects link-layer errors, with strength comparable to ICRC; IPv4 checksum: Verifies the IP header; UDP checksum: End-to-end verification of the UDP payload (including RoCE messages).

[0035] With the existing three-layer check protection, the ICRC of the RoCE layer is redundant. This solution proposes to disable real-time calculation and verification of ICRC in a controlled environment in exchange for performance improvement.

[0036] Figure 2 This is a schematic diagram of the RoCE message layered encapsulation structure provided in an embodiment of the present invention. Figure 2 As shown, the frame composition hierarchy of a standard RoCE Ethernet message is fully displayed from top to bottom. The message is composed of the Ethernet header, IP header, UDP header, IB transport header, application payload, ICRC check field, and Ethernet FCS field concatenated in sequence. Among them, ICRC is a RoCE protocol-specific check field, and FCS is a check field built into the Ethernet link layer. This frame structure is the protocol basis for optimizing the ICRC check logic in this invention.

[0037] Figure 3 This is a schematic diagram illustrating the location of various verification fields in a RoCE message provided in an embodiment of the present invention. For example... Figure 3 As shown, based on the hexadecimal data of the captured packets, the actual storage locations of four types of checksums in the packets are marked: IPv4 header checksum, UDP checksum, IB ICRC, and Ethernet FCS. This intuitively demonstrates that RoCE packets natively possess multi-layer protocol check fields, providing data basis for the present invention to disable ICRC and reuse the three inherent check layers to achieve packet error detection.

[0038] In some possible implementations, the peer node may not support disabling ICRC. To achieve device compatibility, the method in this embodiment further includes: when it is determined that the peer node does not support disabling the invariant cyclic redundancy check, performing the calculation and padding of the invariant cyclic redundancy check, and retaining the field position corresponding to the invariant cyclic redundancy check.

[0039] The padding process refers to the operation of writing the calculated verification result into a fixed field of the message. The field position can be understood as the fixed byte range reserved for ICRC in the RoCE message frame structure.

[0040] Specifically, after the local device detects that the peer node is incompatible with the ICRC shutdown function, it can restore the original processing logic. That is, when sending a message, it calculates the checksum and fills the fields normally, while retaining the original placeholders for the ICRC field inside the message. However, when receiving, it can still skip the ICRC check and rely solely on the underlying verification mechanism. In this way, it automatically supports older peer nodes that do not support disabling ICRC through a degradation compatibility mechanism.

[0041] S120. Store the data to be transmitted into the first memory area, generate a protocol header corresponding to the data to be transmitted, and store the protocol header into the second memory area.

[0042] The data to be transmitted refers to the service payload data that upper-layer business applications need to send out via the RoCE link. These upper-layer applications can be databases, AI training, storage services, etc. The first memory region can be an independent physical memory space allocated by the operating system. The protocol header can be understood as a combined header that sequentially encapsulates the Ethernet header, IP header, UDP header, and RoCE transport header. The second memory region refers to another memory space whose address is not contiguous with the first memory region.

[0043] Specifically, when a local device needs to send data to be transmitted via RoCE, the data to be transmitted can be written into the first memory area, and the protocol stack generates a matching protocol header based on the link parameters of the two communicating parties, and then stores the generated protocol header separately in the second memory area.

[0044] For example, 1. The user-space application (i.e., the next-level business application) prepares the data payload (business payload data), which is located in a certain memory segment; 2. The protocol stack dynamically generates UDP / IP / Ethernet headers, which are located in another memory segment.

[0045] S130. Construct a distributed aggregation list based on the first memory region and the second memory region.

[0046] The scattered aggregation list can be understood as an index list that records multiple discrete memory addresses and length information. This list is used to guide the network card to the data storage location.

[0047] Specifically, the local device extracts the address and length parameters of the first and second memory regions, and completes the creation of the scatter-aggregate list based on the information of these two types of parameters.

[0048] S140. According to the distributed aggregation list, read the data to be transmitted and the protocol message header from the first memory region and the second memory region respectively, assemble them into a RoCE message and send it to the peer node.

[0049] Among them, assembling RoCE messages refers to combining the separately stored message headers and the data to be transmitted into a complete message that conforms to the protocol specifications.

[0050] Specifically, the hardware side refers to the location information recorded in the distributed aggregation list, reads the corresponding content from two memory locations respectively, splices them into a complete RoCE message, and sends it to the peer node.

[0051] In some practical applications, existing technologies present difficulties in capturing RoCE packets. Standard RoCE network cards directly write the payload into memory without forwarding the original packets to the CPU. Packet capture and analysis require relying on the bypass function of specific network cards, which varies across manufacturers, resulting in inconsistent tools and cumbersome operations.

[0052] To address the aforementioned issues, embodiments of the present invention further include: in a message capture scenario, caching the complete RoCE message; and in the message parsing phase, performing the calculation and verification processing of the invariant cyclic redundancy check.

[0053] Among these, packet capture scenarios can include operations and maintenance packet capture, and fault location scenarios involving capturing packets from the entire network. The packet parsing stage refers to the processing stage in which upper-layer software disassembles and analyzes cached packets.

[0054] Specifically, when the packet capture function is enabled, all RoCE packets flowing through the local machine can be fully cached. Subsequently, after entering the packet parsing process, ICRC calculations and verifications are performed separately on the cached packets. This eliminates the need for verification calculations during normal transmission and completes verification in faulty packet capture scenarios, thus balancing transmission performance and fault diagnosis requirements.

[0055] In this embodiment of the invention, while ensuring that ETH FCS, IPv4 checksum, and UDP checksum are enabled, real-time calculation and verification of the RoCE layer ICRC are disabled. RoCE throughput can be increased by 40%, and latency reduced by 20%. In packet capture or debugging scenarios, not verifying ICRC in real time, but delaying the calculation to the subsequent analysis stage, can significantly improve packet capture performance, achieving a 40% improvement without writing to disk. Furthermore, it supports interoperability with nodes that do not support ICRC disabling, calculating ICRC on demand during transmission and skipping verification during reception, achieving compatibility with traditional RoCE devices and supporting gradual network upgrades.

[0056] Traditional RoCE communication performs real-time calculation and verification of Invariant Cyclic Redundancy Check (CRC) by default throughout the entire process, which continuously consumes CPU computing power. The technical solution of this invention configures a policy that allows disabling CRC and negotiates to disable this check function when establishing a link with the peer node, reusing the original check functions of Ethernet, IP, and UDP to achieve message integrity judgment, effectively reducing CPU usage and improving RoCE message throughput performance. Traditional message encapsulation requires the CPU to copy discrete service data and protocol headers to contiguous memory before sending them to the network card. The technical solution of this invention stores the data to be transmitted and the protocol headers in two independent memory locations, constructs a distributed aggregation list, and then uses hardware to read data across regions and assemble messages according to the list, eliminating the CPU memory copying operation and optimizing message sending efficiency.

[0057] Figure 4 This is a flowchart illustrating another high-performance software RoCE data transmission method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the process of constructing the distributed aggregation list and reading the data to be transmitted and the protocol header. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 4 As shown, the method specifically includes the following steps: S410. Configure a policy that allows disabling the invariant cyclic redundancy check (CRRC), and negotiate to disable the CRRC based on the policy when establishing a RoCE connection with the peer node.

[0058] S420. Store the data to be transmitted into the first memory area, generate a protocol header corresponding to the data to be transmitted, and store the protocol header into the second memory area.

[0059] S430. Determine the memory address and corresponding length of the data to be transmitted based on the first memory region to generate the first entry.

[0060] S440. Determine the memory address and corresponding length of the message header based on the second memory region to generate the second entry.

[0061] S450. Determine the memory address and corresponding length of the message tail based on the second memory region to generate a third entry.

[0062] S460. Based on the first entry, the second entry, and the third entry, construct the distributed aggregation list.

[0063] Before introducing the specific implementation steps of this embodiment, the existing technology and the concept of this embodiment will be explained. Traditional software protocol stacks, when constructing RoCE packets, need to concatenate the UDP / IP / Ethernet header and payload into a contiguous memory block before sending it via the network card, resulting in frequent memory copying. In short, traditional software protocol stacks require repeated data copying when constructing RoCE packets, wasting CPU resources and easily causing CPU cache invalidation.

[0064] This invention utilizes the Scatter-Gather Direct Memory Access (SG-DMA) function, commonly supported by modern network interface cards (NICs), enabling the NIC to automatically reassemble packets. Instead of using the CPU to assemble data into contiguous memory, the header and payload are treated as independent memory segments, and a corresponding scatter list is submitted to the NIC. The NIC hardware then automatically reads data from multiple scattered memory blocks and synthesizes them into a complete data packet for transmission, avoiding CPU involvement in copying.

[0065] The entries can be single record units within a scattered aggregate list, with each entry binding the address and length information of an independent memory segment.

[0066] Specifically, the address and length parameters of the first memory region are read and organized to generate the first entry. The message header parameters are split from the second memory region to generate the second entry, and the message tail parameters are split to generate the third entry. The information of the three entries is summarized and integrated to generate a complete scattered aggregated list.

[0067] For example, construct a Scatter-Gather List, which includes: Item 1: payload memory address + length Item 2: Message header memory address + length Item 3: Message tail memory address + length Furthermore, the Scatter-Gather List is submitted to the network card driver.

[0068] S470. Based on the distributed aggregation list, the network interface card (NIC) uses the distributed aggregation direct memory access function to read the data to be transmitted and the protocol header according to the first entry, the second entry, and the third entry, to assemble them into a RoCE message and send it to the peer node.

[0069] Specifically, the network card reads and sends data directly from multiple non-contiguous memory regions via Direct Memory Access (DMA), without requiring CPU copying. This approach can also be used for Transmission Control Protocol (TCP) - TCP Segmentation Offload (TSO) to send Internet Wide Area RDMA Protocol (iWARP) messages.

[0070] Figure 5 This is a schematic diagram illustrating the principle of assembling messages using distributed aggregation descriptors for discrete memory, as provided in an embodiment of the present invention. For example... Figure 5 As shown in the figure, this diagram illustrates the storage and packet sending architecture adopted by the present invention. The packet header, service payload, and ICRC data are stored in independent discrete memory spaces, and three scattered aggregation descriptors point to three memory segments respectively. The network card directly reads the data according to the descriptors based on SG-DMA and hardware assembles the complete packet. The protocol stack does not need to copy multiple data segments to the same contiguous memory, eliminating the CPU memory copy overhead.

[0071] In this embodiment of the invention, the packet header and payload are separated, and the network interface card (NIC) performs the splicing and fragmentation, avoiding CPU access to payload memory. This reduces CPU memory access and cache misses, and the single-core transmission capacity can reach 100Gbps.

[0072] Furthermore, in cloud computing scenarios, cloud service providers find it difficult to fully configure dedicated RDMA hardware network cards, and a large number of general-purpose cloud virtual machines cannot natively carry hardware RoCE capabilities. When developers want to debug RoCE-related business programs using virtual machines, they are constrained by hardware barriers and find it difficult to quickly complete application deployment and functional verification.

[0073] The technical solution of this invention does not rely on dedicated RDMA hardware and can be deployed on cloud virtual machines equipped with ordinary network cards. This solution disables ICRC calculations as needed after connection negotiation, reuses existing Ethernet, IPv4, and UDP checksums to ensure packet reliability, and reduces CPU checksum overhead. Simultaneously, it separates the storage payload and packet header, and directly reads discrete memory data through a distributed aggregation list in conjunction with SG-DMA, eliminating data copying overhead. These two optimizations improve the performance deficiencies of traditional RXE, enabling high-performance RoCE transmission in ordinary cloud virtual machines, facilitating RoCE application debugging directly in virtual machines for developers.

[0074] Traditional RoCE communication performs real-time calculation and verification of Invariant Cyclic Redundancy Check (CRC) by default throughout the entire process, which continuously consumes CPU computing power. The technical solution of this invention configures a policy that allows disabling CRC and negotiates to disable this check function when establishing a link with the peer node, reusing the original check functions of Ethernet, IP, and UDP to achieve message integrity judgment, effectively reducing CPU usage and improving RoCE message throughput performance. Traditional message encapsulation requires the CPU to copy discrete service data and protocol headers to contiguous memory before sending them to the network card. The technical solution of this invention stores the data to be transmitted and the protocol headers in two independent memory locations, constructs a distributed aggregation list, and then uses hardware to read data across regions and assemble messages according to the list, eliminating the CPU memory copying operation and optimizing message sending efficiency.

[0075] Figure 6 This is a schematic diagram of a high-performance software RoCE data transmission device provided in an embodiment of the present invention. Figure 6 As shown, the device includes: The policy configuration module 610 is used to configure a policy that allows the invariant cyclic redundancy check to be turned off, and to negotiate the shutdown of the invariant cyclic redundancy check based on the policy when establishing a RoCE connection with the peer node. The data caching module 620 is used to store the data to be transmitted into a first memory area, generate a protocol header corresponding to the data to be transmitted, and store the protocol header into a second memory area. List building module 630 is used to build a distributed aggregated list based on the first memory region and the second memory region; The message transceiver module 640 is used to read the data to be transmitted and the protocol message header from the first memory region and the second memory region respectively according to the distributed aggregation list, so as to assemble them into a RoCE message and send it to the peer node.

[0076] Traditional RoCE communication performs real-time calculation and verification of Invariant Cyclic Redundancy Check (CRC) by default throughout the entire process, which continuously consumes CPU computing power. The technical solution of this invention configures a policy that allows disabling CRC and negotiates to disable this check function when establishing a link with the peer node, reusing the original check functions of Ethernet, IP, and UDP to achieve message integrity judgment, effectively reducing CPU usage and improving RoCE message throughput performance. Traditional message encapsulation requires the CPU to copy discrete service data and protocol headers to contiguous memory before sending them to the network card. The technical solution of this invention stores the data to be transmitted and the protocol headers in two independent memory locations, constructs a distributed aggregation list, and then uses hardware to read data across regions and assemble messages according to the list, eliminating the CPU memory copying operation and optimizing message sending efficiency.

[0077] In some possible implementations, the policy configuration module 610 includes: The configuration settings submodule is used to set configuration items that allow disabling the invariant cyclic redundancy check via kernel module parameters or user-space configuration files.

[0078] In some possible implementations, the policy configuration module 610 includes: The negotiation and interaction submodule is used to send a connection request carrying a close support flag to the peer node; The capability determination submodule is used to determine, based on the connection request, whether the peer node supports disabling the invariant cyclic redundancy check. The verification shutdown submodule is used to disable the real-time calculation and real-time verification of the invariant cyclic redundancy check when the peer node supports disabling the invariant cyclic redundancy check.

[0079] In some possible implementations, the negotiation interaction submodule is also used for: If it is determined that the peer node does not support disabling the invariant cyclic redundancy check, the calculation and filling process of the invariant cyclic redundancy check is triggered, and the field position corresponding to the invariant cyclic redundancy check is preserved.

[0080] In some possible implementations, the message transceiver module 640 further includes: The send / receive verification submodule is used to stop calculating the field corresponding to the invariant cyclic redundancy check when sending a message, provided that the invariant cyclic redundancy check is disabled. The integrity determination submodule is used to determine the integrity of received messages based on Ethernet frame checksum, IP checksum, and UDP checksum.

[0081] In some possible implementations, the list construction module 630 includes: The entry generation submodule is used to determine the memory address and corresponding length of the data to be transmitted based on the first memory region to generate a first entry, determine the memory address and corresponding length of the message header based on the second memory region to generate a second entry, and determine the memory address and corresponding length of the message tail based on the second memory region to generate a third entry. The list assembly submodule is used to construct the distributed aggregated list based on the first entry, the second entry, and the third entry.

[0082] In some possible implementations, the message transceiver module 640 includes: The hardware reading submodule is used to read the data to be transmitted and the protocol header according to the first entry, the second entry and the third entry through the network card's distributed aggregation direct memory access function.

[0083] In some possible implementations, the message transceiver module 640 further includes: The message caching submodule is used to cache complete RoCE messages in message capture scenarios; The post-verification submodule is used to perform the calculation and verification processing of the invariant cyclic redundancy check during the message parsing stage.

[0084] The high-performance software RoCE data transmission device provided in the embodiments of the present invention can execute the high-performance software RoCE data transmission method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0085] Figure 7This is a schematic diagram of the structure of an electronic device for implementing the high-performance software RoCE data transmission method according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0086] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0087] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the high-performance software RoCE data transfer method.

[0089] In some embodiments, the high-performance software RoCE data transfer method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the high-performance software RoCE data transfer method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the high-performance software RoCE data transfer method by any other suitable means (e.g., by means of firmware).

[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific integrated circuits (ASICs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0095] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0096] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A high-performance software RoCE data transmission method, characterized in that, include: Configure a policy that allows disabling Invariant Cyclic Redundancy Check (CRC), and negotiate to disable CRC based on the policy when establishing a RoCE connection with the peer node. The data to be transmitted is stored in the first memory area, a protocol header corresponding to the data to be transmitted is generated, and the protocol header is stored in the second memory area. Construct a distributed aggregation list based on the first memory region and the second memory region; According to the distributed aggregation list, the data to be transmitted and the protocol message header are read from the first memory region and the second memory region respectively, to assemble them into a RoCE message and send it to the peer node.

2. The method according to claim 1, characterized in that, The configuration allows for policies that disable invariant cyclic redundancy check (CRC) including: Configure the option to disable the invariant cyclic redundancy check via kernel module parameters or user-space configuration files.

3. The method according to claim 1, characterized in that, When establishing a RoCE connection with the peer node, negotiating to disable the invariant cyclic redundancy check based on the policy includes: Send a connection request carrying a close support flag to the peer node; Based on the connection request, determine whether the peer node supports disabling the invariant cyclic redundancy check; If the peer node supports disabling the invariant cyclic redundancy check, then the real-time calculation and real-time verification of the invariant cyclic redundancy check shall be disabled.

4. The method according to claim 3, characterized in that, Also includes: If it is determined that the peer node does not support disabling the invariant cyclic redundancy check, the calculation and filling process of the invariant cyclic redundancy check is performed, and the field position corresponding to the invariant cyclic redundancy check is retained.

5. The method according to claim 1, characterized in that, Also includes: When the invariant cyclic redundancy check is disabled, the field corresponding to the invariant cyclic redundancy check is no longer calculated when sending a message; When receiving a message, the integrity of the message is determined based on Ethernet frame checksum, IP checksum, and UDP checksum.

6. The method according to claim 1, characterized in that, The construction of the distributed aggregation list based on the first memory region and the second memory region includes: The memory address and corresponding length of the data to be transmitted are determined based on the first memory region to generate the first entry; The memory address and corresponding length of the message header are determined based on the second memory region to generate the second entry; The memory address and corresponding length of the message tail are determined based on the second memory region to generate the third entry; The decentralized aggregation list is constructed based on the first entry, the second entry, and the third entry.

7. The method according to claim 6, characterized in that, The step of reading the data to be transmitted and the protocol header from the first memory region and the second memory region respectively according to the distributed aggregation list includes: The network interface card (NIC) uses distributed aggregation direct memory access (DRAM) to read the data to be transmitted and the protocol header based on the first entry, the second entry, and the third entry.

8. The method according to claim 1, characterized in that, Also includes: In message capture scenarios, cache complete RoCE messages; During the message parsing phase, the calculation and verification process of the invariant cyclic redundancy check is performed.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the high-performance software RoCE data transmission method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the high-performance software RoCE data transmission method according to any one of claims 1-8.

Images