Method and apparatus for transmitting a packet

Abstract

This application provides a message transmission method applied to a first device and a second device connected via a queue pair (QP). The queue pair transmits messages through multiple paths. The method includes: the sending device locally storing a first mapping table and a sending queue. The first mapping table stores the identifiers of messages received by the receiving device, and the sending queue stores messages to be sent. When a first message to be sent is a message that needs to be sent in order, the sending device can determine whether the preceding message of the first message has been received by the receiving device based on the message identifier stored in the first mapping table. If not, the sending device buffers the first message and continues to send subsequent messages that do not need to be sent in order. This method can avoid blocking the transmission of subsequent messages that do not need to be sent in order and can fully utilize the bandwidth of the Internet.

Description

Methods and apparatus for message transmission Technical Field

[0001] This application relates to the field of communications, and in particular to a method and apparatus for message transmission. Background Technology

[0002] With the integration of fifth-generation mobile communication technology (5G), big data, the Internet of Things (IoT), and artificial intelligence (AI) into all aspects of human society, it is foreseeable that humanity will enter an intelligent society based on the digital world, characterized by ubiquitous sensing, interconnection, and intelligence. High-performance data center computing power is becoming a new productive force, and the dimension of data centers is shifting from resource scale to computing power scale. The concept of computing power centers has been widely accepted in the industry. As data centers evolve into computing power centers, the network is the most crucial component for achieving high computing power. Improving network performance can significantly improve the energy efficiency ratio of data center computing power.

[0003] High-performance data centers demand high throughput and low latency in their networks. To achieve these two key metrics, the industry generally uses Remote Direct Memory Access (RDMA) technology to replace the traditional Transmission Control Protocol (TCP) technology, thereby reducing latency and lowering the utilization rate of the data center's central processing unit (CPU).

[0004] Currently, InfiniBand (IB) and RDMA over Converged Ethernet (RoCE) are the mainstream RDMA technologies widely used in high-performance data center interconnects. IB and RoCE are network protocols designed for RDMA, guaranteeing lossless network performance at the hardware level, offering extremely high bandwidth and extremely low latency. However, both IB and RoCE only support forced (fence) and strong-ordering message ordering in their interconnection message ordering methods. Forced ordering requires confirmation that preceding messages have been fully received before sending a forced-ordered message, increasing network latency; strong-ordering requires single-path transmission in the interconnection network, failing to fully utilize network bandwidth.

[0005] The existing message ordering methods in RDMA technology limit the data center from achieving its full performance in scenarios with extremely low latency requirements and extremely high network bandwidth. Therefore, how to provide a new message ordering method to make full use of the multipath transmission of the Internet and maximize the network's transmission performance has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a message transmission method that can achieve message order preservation while making full use of the multipath transmission of the Internet and maximizing the transmission performance of the network.

[0007] Firstly, a method for message transmission is provided, applied to a first device. This method can be executed by the first device, or by circuitry configured within the first device; this application does not limit the execution in this regard. For convenience, the following description uses execution by the first device as an example. A queue pair (QP) connection is established between the first device and a second device. This QP connection enables the sending and receiving of messages. Specifically, the QP connection transmits messages through at least one Transport Group (TPG).

[0008] For example, the first device acts as the message sender and the second device acts as the message receiver; or, the first device acts as the message receiver and the second device acts as the message sender. For ease of description, the following explanation will use the example of the first device acting as the message sender and the second device acting as the message receiver.

[0009] The first device stores a first mapping table and a sending queue. The first mapping table includes identifiers of messages that have been received by the second device, and the sending queue includes messages to be sent. The message transmission method includes: the first device obtaining a first message from the sending queue, the first message being a pre-order message; when the identifier of at least one preceding message of the first message is not in the first mapping table, the first device obtaining and sending a non-pre-order message following the first message.

[0010] Based on the above technical solution, when a first message to be sent is a message that needs to be sent in order, the sending device can determine whether the preceding message of the first message has been received by the other end according to the message identifier stored in the first mapping table. If not, it continues to send the subsequent messages that do not need to be sent in order, which can avoid blocking the transmission of subsequent messages that do not need to be sent in order and make full use of the bandwidth of the Internet.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, when the identifiers of the preceding messages of the first message are all in the first mapping table, the first device sends the first message.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first message is a transaction layer message, and the first message includes a transaction layer header (TA header). The TA header includes a first sequence number (SN), a first indication information, and a second indication information. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the order preservation requirement of the first message.

[0013] Based on the above technical solution, the first message can be a transaction layer message, achieving ordered transmission of transaction layer messages while fully utilizing the bandwidth of the Internet. Specifically, the transaction layer message includes a transaction layer header (TA header), which includes a first serial number (SN) for identifying the message sequence, so that the transport layer of the first device can determine the order of the first message among multiple messages based on the first SN. The TA header also includes an indication of the transfer group (TPG) corresponding to the first message, so that the transport layer of the first device can determine to transmit the first message within the TPG corresponding to the first message. The TA header also includes second indication information indicating the ordering requirement of the first message, so that the transport layer of the first device can determine whether the first message is a message that needs to be transmitted in order.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the first device sending the first message includes: the transaction layer of the first device sending the first message to the transport layer of the first device; and the transport layer of the first device sending the first message to the transport layer of the second device.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the first message is a transport layer message, which includes a transaction layer header (TA) and a transport layer header (TP). The TA header contains a first sequence number (SN), first indication information, and second indication information. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the ordering requirement of the first message. The TP header contains a second SN of the first message.

[0016] Based on the above technical solution, the specific transmission process of the first message can be as follows: the transaction layer of the first device transmits the first message to the transport layer of the first device, and the transport layer of the first device then transmits the first message to the transport layer of the second device. That is, a connection is established between the transport layers of the two devices, and the transaction layer is unaware of the connection establishment.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, after the first device sends the first message, the method further includes: the first device saving the first SN to the first mapping table.

[0018] Based on the above technical solution, after the first device successfully sends a message, it can save the SN of the message to the first mapping table so as to determine whether subsequent messages to be sent can be sent directly.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, when the identifier of at least one preceding message of the first message is not in the first mapping table, the method further includes: the first device caching the first message in the cache of the first device.

[0020] Based on the above technical solution, if the preceding messages of the first message are not all received by the second device, the first device can cache the first message in the cache, and when the sending conditions are met later, it can directly retrieve it from the cache and send it.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the first device receiving confirmation information from the second device, the confirmation information indicating that the second message has been received by the second device; the first device saving the identifier of the second message to the first mapping table and updating the first mapping table; and the first device determining whether to send the first message based on the updated first mapping table.

[0022] Based on the above technical solution, after the first device receives the confirmation information from the second device, it can determine that the second message has been received by the second device based on the confirmation information. Therefore, the first device can save the identifier of the second message to the first mapping table, update the first mapping table, and re-determine whether the first message can be sent based on the updated first mapping table. The first device promptly determines whether the message can be sent based on the feedback from the second device, ensuring the timeliness of message transmission.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the first device obtaining and sending a non-orderly message following the first message includes: the first device obtaining a third message following the first message from the sending queue; the first device determining that the third message is a non-orderly message based on the ordering requirement carried in the transaction layer header (TA) of the third message; and the first device sending the third message.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the context of the sending queue includes the following information: a first index, a first valid flag, and a first queue index, wherein the first index is used to indicate the first mapping table, the first valid flag is used to indicate whether there are any ordered messages to be sent in the sending queue, the first queue index is used to indicate the first queue, and the first queue is used to buffer unsent ordered messages.

[0025] Secondly, a message transmission method is provided, applied to a second device. This method can be executed by the second device, or by circuitry configured within the second device; this application does not limit the execution in this regard. For convenience, the following description uses the execution by the second device as an example. A queue pair (QP) connection is established between the second device and the first device. This QP connection enables the sending and receiving of messages. Specifically, the QP connection transmits messages through at least one Transport Group (TPG).

[0026] For example, the first device acts as the message sender and the second device acts as the message receiver; or, the first device acts as the message receiver and the second device acts as the message sender. For ease of description, the following explanation will use the example of the first device acting as the message sender and the second device acting as the message receiver.

[0027] The second device stores a second mapping table, which includes identifiers of messages that have been received by the second device. The message transmission method includes: the second device receiving a first message from the first device; the second device storing the identifier of the first message in the second mapping table and updating the second mapping table; the first message being a pre-order message; when the identifiers of the preceding messages of the first message are all in the second mapping table, the second device processes the first message and its preceding messages; when the identifier of at least one preceding message of the second message is not in the second mapping table, the second device buffers the first message and receives and processes non-pre-order messages following the first message.

[0028] Based on the above technical solution, after the receiving device receives the first message with order requirements, if not all the preceding messages of the first message have been received, the first message can be buffered and the subsequent non-order messages can be received. This can avoid blocking the processing of subsequent messages that do not require order transmission and make full use of the bandwidth of the Internet.

[0029] For a description of the first message, please refer to the description of the first message in the first aspect; it will not be repeated here.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, after the second device caches the first message, the method further includes: the second device receiving and processing the second message, the second device storing the identifier of the second message in the second mapping table, updating the second mapping table, wherein the second message is not an ordered message; and the second device determining whether to process the first message based on the updated second mapping table.

[0031] Based on the above technical solution, after the second device receives other messages, it can save the identifier of the received message to a second mapping table, update the second mapping table, and re-determine whether the first message can be processed based on the updated second mapping table. The second device promptly determines whether a message can be processed based on the message reception status, ensuring the timeliness of message processing.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the second device stores a receive queue, the context of which includes the following information: a second index, a second valid flag, and a second queue index. The second index is used to indicate the second mapping table, the second valid flag is used to indicate whether there are received ordered messages in the receive queue, the second queue index is used to indicate the second queue, and the second queue is used to buffer unprocessed ordered messages.

[0033] Thirdly, a message transmission apparatus is provided, comprising: a storage module for storing a program; and a processing module for executing the program stored in the storage module. When the program stored in the storage module is executed, the processing module executes the methods provided in the above aspects.

[0034] Fourthly, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including methods for performing the methods provided in the above aspects.

[0035] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the methods provided in the above aspects.

[0036] In a sixth aspect, a chip is provided, the chip including a processing module and a communication interface, the processing module reading instructions stored in a memory through the communication interface for executing the methods provided in the above aspects.

[0037] Optionally, as one implementation, the chip may further include a storage module storing instructions, and the processing module is used to execute the instructions stored in the storage module. When the instructions are executed, the processing module is used to perform the methods provided in the above aspects.

[0038] In a seventh aspect, a chip is provided, the chip including a first device for performing the method provided in the first aspect and a second device for performing the method provided in the second aspect.

[0039] Eighthly, a computer device is provided, which includes the chip shown in the seventh aspect. For example, the computer device includes, but is not limited to, switches or servers in a data center.

[0040] Ninthly, a terminal device is provided, the terminal device including the chip shown in the seventh aspect. For example, the terminal device includes, but is not limited to, terminals such as mobile phones and vehicles.

[0041] In a tenth aspect, a message transmission system is provided, the system comprising a first device for performing the method provided in the first aspect and a second device for receiving messages. Attached Figure Description

[0042] Figure 1(a) is a schematic diagram of the structure of the computer device provided in the embodiment of this application.

[0043] Figure 1(b) is a schematic diagram of a data center provided in an embodiment of this application.

[0044] Figure 2 is a schematic diagram of the general bus protocol message format provided in the embodiments of this application.

[0045] Figure 3(a) is a schematic diagram of an application scenario structure provided by an embodiment of this application.

[0046] Figure 3(b) is a schematic diagram of a chip receiver provided in this application.

[0047] Figure 4(a) and (b) are schematic diagrams of the order-preserving method.

[0048] Figure 5 is a schematic flowchart of a message transmission method provided in an embodiment of this application.

[0049] Figure 6 is a schematic flowchart of another message transmission method provided in an embodiment of this application.

[0050] Figure 7 is a schematic diagram of a data structure constructed by the sending end according to an embodiment of this application.

[0051] Figure 8 is a schematic flowchart of another message transmission method provided in an embodiment of this application.

[0052] Figure 9 is a schematic diagram of a data structure constructed by a receiving end according to an embodiment of this application.

[0053] Figure 10 shows a schematic diagram of the structure of a message transmission device 1000 provided in an embodiment of this application.

[0054] Figure 11 shows a schematic diagram of the structure of a chip system 1100 provided in an embodiment of this application.

[0055] Figure 12 schematically illustrates a conceptual partial view of a computer program product provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0057] In many applications, the computer devices that deploy the application need to access data to enable the application's functionality. For example, computer devices deploying database applications need to perform a large amount of data access to update data in the database or to respond to data query requests and return query results to the user. Similarly, computer devices deploying web applications need to perform a large amount of data access to return requested content to the user.

[0058] Computer equipment can be servers, switches, or terminals. Terminals include, but are not limited to, user devices such as desktop computers, laptops, and smartphones. For ease of understanding, the structure of computer equipment is described below.

[0059] Referring to the structural diagram of the computer device shown in Figure 1(a), the computer device includes a processor 101, an input output device (IO device) 102, a memory 103, a cache 104, a memory management unit (MMU) 105, an input output management unit (IOMMU) 106, external storage 107, and a bus 108.

[0060] Processor 101 includes at least one core. This core is also called a computing engine. Each core can execute tasks independently. When processor 101 includes multiple cores, tasks from applications can be divided, allowing applications to fully utilize multiple cores and execute more tasks within a given time. In this embodiment, processor 101 can be a main processor, such as a Central Processing Unit (CPU).

[0061] Input / output device 102 refers to a hardware device capable of inputting and / or outputting data. Input / output device 102 can be divided into input devices and output devices. Input devices may include devices such as a mouse, keyboard, joystick, stylus, and microphone, while output devices may include devices such as a monitor and speakers.

[0062] Memory 103, also known as internal memory or main memory, is used to temporarily store the data processed by the processor 101. Furthermore, memory 103 is also used to temporarily store data exchanged with secondary memory 107. Memory 103 can typically be implemented using storage media such as dynamic random access memory (DRAM) or static random access memory (SRAM).

[0063] Cache 104 (in this embodiment, it refers to the processor cache, such as the CPU cache) is a component used to reduce the average time required for the processor 101 to access memory 103. Referring to Figure 2, in the pyramid-shaped memory architecture, cache 104 is located in the second layer from the top, second only to the registers of the processor 101 (not shown in Figure 1(a)) and above memory 103 (memory 103 is located in the third layer from the top). Typically, the capacity of cache 104 is much smaller than that of memory 103, but its access speed can be close to the frequency of the processor 101.

[0064] The memory management unit 105 is a type of computer hardware used to process data access requests. Specifically, the memory management unit 105 maps virtual addresses (VAs) in data access requests. Specifically, the memory management unit 105 can intercept data access requests issued by the processor 101's core and map (or translate) the virtual addresses in the data access requests into physical addresses (PAs), so that memory 103 can be accessed based on those physical addresses.

[0065] The input / output memory management unit 106 is essentially a memory management unit. Similar to the memory management unit 105, which maps the virtual address visible to the processor 101 to the physical address, the input / output memory management unit 106 is used to map the virtual address (also called the device address or I / O address) visible to the input / output device 102 to the physical address.

[0066] External storage 107, also known as secondary storage or auxiliary storage, is typically used for persistent data storage. For example, external storage 107 can persistently store computational data from processor 101. Even in the event of a power failure, data already written to external storage 107 can still be preserved, preventing data loss. In specific implementations, external storage 107 includes at least one non-volatile memory 1071. When external storage includes multiple non-volatile memories, these multiple non-volatile memories can be of the same type or different types. For example, in the example in Figure 1(a), external storage 107 can include two types of non-volatile memories, such as storage class memory (SCM) and solid state drive (SSD).

[0067] Bus 108 is used to connect the various functional components of a computer device. Bus 108 is a common communication trunk for transmitting information between the various functional components of a computer device. Bus 108 can be a transmission line bundle formed by wires. Depending on the objects being connected, bus 108 can also be divided into internal bus and external bus.

[0068] The internal bus uses an internal bus protocol to transmit information. The internal bus protocol includes a bus protocol for accessing the memory space of the computer device. The external bus uses an external bus protocol to transmit information. The external bus protocol includes a bus protocol for accessing the external storage space of the computer device. Here, memory space refers to the address space of the main memory, and external storage space refers to the address space of the secondary storage.

[0069] In some embodiments, the internal bus protocol includes, but is not limited to, the Peripheral Component Interconnect (PCI) bus, the PCI Express (PCI-E) protocol, the Intel™ Quick Path Interconnect (QPI) protocol, and the Unified Bus (UB) protocol. The external bus protocol includes, but is not limited to, the Small Computer System Interface (SCSI) protocol or the Serial Attached SCSI (SAS) protocol.

[0070] It should be noted that the computer device shown in Figure 1(a) is illustrated using external storage 107 as a remote external storage unit. As shown in Figure 1, external storage 107 includes a network interface card (NIC) 1072. This NIC 1072 can be, for example, a smart NIC (network interface card) (i.e., a network adapter card). External storage 107 accesses the network through this NIC 1072, and then connects to other components of computer device 101 through the network. The network can be a wired communication network, such as a fiber optic communication network, or a wireless communication network, such as a wireless local area network (WLAN) or a fifth-generation (5G) mobile communication network.

[0071] In some possible implementations, the computer device's external storage 107 can also be local external storage, and other components of the computer device, such as the processor 101, can be connected to the aforementioned local external storage via bus 108. In other possible implementations, the computer device can include both remote external storage and local external storage. Furthermore, the embodiments of this application can be applied to centralized storage or distributed storage scenarios, and this embodiment does not limit them.

[0072] This application primarily relates to cross-network message transmission, and exemplaryly, it is applied to server clusters requiring cross-network communication, such as the data center shown in Figure 1(b). The internal structure of the switch or server shown in Figure 1(b) is as described in Figure 1(a) above. The computer equipment discussed below includes, but is not limited to, switches or servers in a data center.

[0073] In addition, the device that transmits messages in this application may be a device that supports the Unified Bus (UB) protocol (or may support other bus protocols, which this application does not limit). The Unified Bus protocol may also be called Lingqu Bus or Unified Bus, a bus protocol standard. This application does not limit the name of the Unified Bus.

[0074] The Universal Bus Protocol breaks down existing protocol barriers and eliminates unnecessary intermediate conversion overhead, thereby achieving extremely low latency. The Universal Bus Protocol defines independent transaction and transport layers. There are connections between transport layers, but no connections between transaction layers. All transactions within a host are carried on a single transport layer. The Universal Bus Protocol includes a transport layer and a transaction layer. The transport layer is responsible for retransmitting lost packets and ensuring reliable transmission, while the transaction layer handles its respective transactions. The transport layer receives packets from the network, removes the transport layer header, and forwards them to the transaction layer.

[0075] The general bus protocol message format is shown in Figure 2. Specifically, the field definitions in the general bus protocol message format are shown in Table 1 below:

[0076] Table 1

[0077]

[0078] Specifically, the interaction interface between the transaction layer and the application in the general bus protocol is called a Jetty. Application messages can be sent to any destination through a single Jetty, and messages can be received from any source through a single Jetty. A Jetty that can only send is defined as (Jetty For Send, JFS); a Jetty that can only receive is defined as (Jetty For Receive, JFR).

[0079] This application is primarily used in cluster computing fields such as data centers and high-performance computing (HPC) to improve the scale of interconnected systems, reduce internal latency, and increase transmission bandwidth. The application system framework of this application is shown in Figure 3(a). Two nodes (the initiator node and the target node shown in Figure 3(a)) transmit data through an interconnection network within the system. Within the initiator node, message processing on the sending side is divided into a transaction layer (TA layer) and a transport layer (TP layer). The TA layer implements message-level hardware and software interaction processing, while the TP layer implements reliable transmission processing of TA message data.

[0080] Multiple Transport Groups (TPGs) are established between the Initiator and Target nodes at the TP transport layer. Each TPG contains multiple Transport Ports (TPs). Load balancing is achieved among the multiple TPs within a TPG. Different TP packets can be transmitted through different network paths. Congestion control is implemented at the TP granularity, and retransmission is used to ensure reliable transmission in case of packet loss. For example, the internal structure of the node in Figure 3(a) can be as shown in Figure 1(a), meaning that the node involved in Figure 3(a) can be a computer device.

[0081] Specifically, the initiating node's working queue (WQ) is a message sending or receiving queue created by the communication software process. For ease of understanding, the following describes the interaction between a process's sending queue on the sending node and a process's receiving queue on the target node:

[0082] (1) The sending node process writes multiple sending tasks into the WQ, generating multiple working queue elements (WQE);

[0083] (2) The TA layer of the sending node retrieves the WQE from the WQ and sends the WQE to the corresponding TPG. Different WQEs within one WQ can be transmitted through different TPGs.

[0084] (3) WQE transmission can achieve load balancing of transmission across multiple TPs within a TPG.

[0085] (4) The target node TP layer implements TPG message assembly. After the message assembly is completed in TA, it requests the receive work queue WQE to complete the message storage and reports the completion queue element (CQE) to the software process to inform the process that the interactive message receiving process is complete.

[0086] It should be understood that the sending and receiving parties in Figure 3(a) are referred to as the initiating node and the target node, which is only an example and does not constitute any limitation on the scope of protection of this application. They can also be referred to as "first node" and "second node", or "first device" and "second device", etc., which are common names.

[0087] For example, the product form involved in this application is a server or network switch chip in a cluster system such as a data center and high-performance computing. This application belongs to the input / output (IO) or interconnect chip subsystem of the relevant chip, which realizes flexible order preservation of the transmission of system interconnect chip messages through hardware logic or software cooperation.

[0088] The chip structure is shown in Figure 3(b), which is a schematic diagram of a chip receiver provided in this application, consisting of three parts: WQ, TA layer and TP layer.

[0089] WQ is the information format exchanged between software processes and hardware logic. On the sending end device side, the software generates the WQ, the hardware obtains, parses, processes, and transmits the corresponding data. On the receiving end device side, the hardware obtains the WQ, stores the received data into memory according to the parsing, and simultaneously writes to the queue to notify the software, which then obtains and performs subsequent processing.

[0090] The Transaction Layer (TA) lies between the transaction software interface and the transport layer. On the sending device side, the transaction layer transforms software-issued commands (via Write-Queued Transaction) into transport transactions and sends them to the transport layer. On the receiving device side, the transaction layer receives the transport layer transactions, transforms them into memory operations (via Write-Queued Transaction) and notifies the application as needed. The transaction layer protocol defines the interaction interface between the transaction layer and the transport layer, the execution flow of various transactions, and how the transaction layer handles transaction ordering, large transaction splitting, and other characteristics.

[0091] The Transport Layer (TP) provides end-to-end transmission services, supporting single-path and multi-path transmission, and features end-to-end congestion control and packet order maintenance. The transport layer also provides two types of transmission services: reliable and unreliable. Reliable services guarantee that data is delivered intact to the other end; when packets are lost, the transport layer is responsible for retransmission. Unreliable services do not guarantee data integrity.

[0092] The following is a detailed description of each module.

[0093] WQ can be a send queue, a receive queue, and a completion queue. The send and receive queues are generated by software and read, parsed, and processed by hardware. The completion queue is generated by hardware and read, parsed, and processed by software. Each queue contains multiple processing units. The send and receive queues correspond to WQE, and the completion queue corresponds to CQE. The software and hardware interact through the data interface between WQE and CQE. WQ is managed through WQ Context (WQC), which contains information such as WQE index and order-preserving bitmap index. These will be described in detail below with reference to specific embodiments, and will not be repeated here.

[0094] The TA layer is used to retrieve the WQC content of the corresponding queue and read the corresponding WQE information based on the queue number. The TA layer's order preservation includes sender-side order preservation TA_TX_ODR and receiver-side order preservation TA_RX_ODR functions.

[0095] In the scenario where the sending device maintains the order bitmap based on the received response message, it also assigns a WQE number to each sent WQE, incrementing the SN number for each sent WQE, and splitting large transaction WQEs to obtain TP WQEs with the number TPMSN, which are then transmitted to TP. This will be described in detail below with reference to specific embodiments, and will not be repeated here.

[0096] In the case of order preservation in the receiving device, the order preservation maintenance on the receiving side is performed based on the received message and the corresponding order preservation flag.

[0097] The TP layer is used to receive TP WQEs formed after the TA layer splits the data. For each TP WQE processed, each TPWQE is sent to a different TPG for transmission within the TP. Within the TPG, it is weighted and then sent to a specific TP. For a given TP, the data corresponding to the TP WQE is split into packets for transmission. In scenarios requiring order preservation at the receiving end, TP-level order preservation is achieved through the sequence number and its order preservation flag in the received packet. TP-level order preservation only supports the receiving end device's TP_RX_ODR order preservation function.

[0098] As can be seen from the interaction flow between the sending queue of a process at the sending node and the receiving queue of a process at the destination node, TPG's multipath transmission fully utilizes bandwidth, but also introduces the issue of message ordering. High-performance data centers demand high throughput and low latency from their networks. To achieve these two key indicators, the industry generally uses RDMA technology to replace traditional TCP technology, reducing latency and lowering CPU utilization in the data center. Currently, IB and RoCE, as mainstream RDMA technologies, are widely used in high-performance data center interconnects. IB and RoCE are network protocols specifically designed for RDMA, guaranteeing network losslessness at the hardware level, and possessing extremely high bandwidth and extremely low latency. However, IB and RoCE protocols only support forced (fence) ordering and strong-ordering for interconnect message ordering. Forced ordering requires confirmation that preceding messages have been fully received before transmission, increasing network latency; strong-ordering requires single-path transmission in the interconnect network, which cannot fully utilize network bandwidth.

[0099] The above text, in conjunction with Figure 2, briefly introduces the scenarios in which this application can be applied. In order to facilitate understanding of the embodiments of this application, some basic concepts involved in this application are briefly explained.

[0100] 1. RDMA: Direct Memory Access (RDMA) moves data directly from one computer to another via a network, quickly transferring data from one system to a remote system's memory without requiring intervention from the operating systems or kernels of both computers. RDMA eliminates the overhead of external memory copying and context switching, thus freeing up memory bandwidth and CPU cycles to improve application performance. Currently, large-scale deployments of RDMA primarily utilize InfiniBand and RoCE. The former is mainly used in high-performance applications, while the latter is more commonly found in internet company data centers.

[0101] RDMA is a transport layer network protocol that uses queue pairs (QPs) in the channel adapter (CA) to send and receive data (as messages) to consumers. Each QP contains a send queue (SQ) and a receive queue (RQ), where the SQ is responsible for sending messages and the RQ is responsible for receiving messages. Each QP's SQ and RQ can be associated with a completion queue (CQ). Each QP has its own connection context to maintain its connection state, and each SQ, RQ, and CQ also has its own queue context to maintain its queue usage state. The consumer refers to the entity that uses the QP to send and receive messages, generally a software process, such as the NoF protocol software.

[0102] 2. Segment Sequence Number (SSN): Transaction layer packets may be quite large, such as 16MB. Multiple transaction layers share a single transport layer. To prevent a single transaction layer packet from occupying the transport layer connection for an extended period, the transaction layer can divide the packet into multiple slices when sending it to the transport layer. For example, a slice may be 64KB. A transaction layer sends only one slice to the transport layer at a time, and each slice includes its corresponding SSN.

[0103] For example, the sending device performs two-level splitting of the transmitted message. The TP Layer receives the TP WQE formed by the TA Layer after splitting. For each TP WQE processed, the transport layer message sequence number (TPMSN) is incremented. Within the TP Layer, each TP WQE is sent to the TP of a different TPG. For a certain TP, the data corresponding to the TP WQE is split into messages for transmission. Each message carries a packet sequence number (PSN). The TP Layer increments the PSN for each message sent.

[0104] 3. Transaction ACK (TAACK): After the receiving side receives a Segment (a Segment is split into multiple packets at the transport layer) and executes it correctly (e.g., correctly reads or writes memory), it returns TAACK to inform the sending side's transaction layer that the segment has been executed correctly. Alternatively, the receiving side can return TAACK to inform the sending side's transaction layer that the message has been successfully received if it successfully receives the message from the sending side.

[0105] 4. Transaction No OK ACK (TANAK): If the receiving side receives a segment and an error occurs (e.g., a page fault occurs when reading or writing memory), it returns TANAK to inform the sending side's transaction layer to retransmit the segment. Alternatively, if the receiving side fails to receive the message from the sending side, it returns TANAK to inform the sending side's transaction layer to retransmit the message.

[0106] 5. Forced (Fence) Ordering: This ensures ordering for the sender and supports multipath transmission. For packets carrying the Fence tag (e.g., packet C shown in Figure 4(a)), it is required that the packets preceding the packet carrying the Fence tag (e.g., packets A and B shown in Figure 4(a)) be completely received by the target. That is, the sender sends the packet only after receiving the acknowledgment (ACK).

[0107] As shown in Figure 4(a), the messages to be sent by the sender include messages A, B, C, and D. Message C carries a Fence flag. The sender can send messages to the target via multiple paths, as shown in Figure 4(a), sending messages A and B simultaneously via paths #1 and #2. Since message C carries a Fence flag, it can only be sent after messages A and B have been completely received by the target. If the sender receives ACKs corresponding to messages A and B respectively (e.g., ACK-A and ACK-B as shown in Figure 4(a)), the sender determines that messages A and B have been completely received by the target and can send message C. Message D is sent after receiving the ACK corresponding to message C (e.g., as shown in Figure 4(a)).

[0108] 6. Strong-Ordering: The receiver maintains the order and does not need to wait for the target to fully receive the data before transmitting. However, both the receiver and the target in the communication require that only one transmission path be established in the Internet.

[0109] As shown in Figure 4(b), the messages to be sent by the sending end include messages A, B, C, and D, among which messages A, B, C, and D carry a Strong-Ordering flag. The sending device transmits messages A, B, C, and D sequentially through a single transmission path to ensure that the receiving device receives the messages in order.

[0110] 7. Ordered Messages: As mentioned above, forced ordering and strong-ordering refer to messages with ordering requirements. In this application, messages with ordering requirements are called ordered messages, and messages without ordering requirements are called non-ordered messages. For example, messages with ordering requirements include, but are not limited to, messages that depend on the reception of other messages. For instance, if message #1 can only be sent if message #2 is successfully received, then message #1 is an ordered message. Similarly, messages without ordering requirements include, but are not limited to, messages that do not depend on the reception of other messages.

[0111] In addition, the following points are provided to facilitate understanding of the embodiments of this application.

[0112] First, in this application, "at least one" refers to one or more, and "more than one" refers to two or more. Furthermore, in the embodiments of this application, "first," "second," and various numerical designations (e.g., "#1," "#2," etc.) are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The sequence numbers of the processes below do not imply an order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that the objects described in this way can be interchanged where appropriate to describe solutions other than those in the embodiments of this application. Moreover, in the embodiments of this application, terms such as "S510" are merely identifiers for descriptive convenience and do not limit the order of execution steps.

[0113] Second, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0114] Third, the term "storage" in the embodiments of this application can refer to storage in one or more memories. These memories can be separate installations or integrated into an encoder, decoder, processor, or communication device. Alternatively, some memories can be separately installed, while others are integrated into the decoder, processor, or communication device. The type of memory can be any form of storage medium, and this application does not limit this.

[0115] Fourth, the term "comprising" (also referred to as "includes", "including", "comprises" and / or "comprising") used in the embodiments of this application, when used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0116] Fifth, the word "if" in the embodiments of this application can be interpreted as meaning "when" or "upon" or "in response to determination" or "in response to detection". Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is detected" can be interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected" or "in response to detection of [the stated condition or event]".

[0117] Sixth, the terminology used in the description of the various examples in the embodiments of this application is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and in the appended claims, the numerical forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0118] Seventh, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0119] The preceding text, with reference to Figure 2, briefly introduced the applicable scenarios for the message transmission method provided in this application, and also introduced the basic concepts involved in this application. Among these basic concepts, Fence ordering and Strong-Ordering were introduced. Specifically, Fence ordering and Strong-Ordering have the following problems:

[0120] (1) Under the Fence ordering mode, all packets before the forced ordering packet must be sent and ACK must be received before the packet can be sent. The forced ordering packet must be sent and ACK must be received before the subsequent packets can be sent. This may block the early transmission of subsequent packets that do not require ordering, wasting the bandwidth of the interconnecting network and increasing the overall transmission delay.

[0121] (2) In the Strong-Ordering mode, only single-path transmission is supported, and the bandwidth of the interconnected network system cannot be fully utilized.

[0122] To address the problems of current order-preserving methods, this application proposes a message transmission method that enables non-blocking and multi-path transmission, reducing overall transmission latency and improving bandwidth utilization in the Internet.

[0123] It should be understood that the message transmission method provided in the embodiments of this application can be applied to computer systems, such as the data center shown in Figure 2.

[0124] It should also be understood that the embodiments shown below do not particularly limit the specific structure of the execution subject of the method provided in the embodiments of this application. As long as the method provided in the embodiments of this application can be implemented by running a program that records the code of the method provided in the embodiments of this application. For example, the execution subject of the method provided in the embodiments of this application can be a device, or a functional module in the device that can call and execute a program.

[0125] Figure 5 is a schematic flowchart of a message transmission method provided in this application. It includes the following steps:

[0126] S510, the first device generates multiple message groups.

[0127] Specifically, the first device can generate multiple message groups for different transactions, where a "transaction" can be understood as a task to be executed by the first device. For example, a message generated for accessing the memory of the second device is called the first message group, and a message generated for sending data to the second device is called the second message group. Each message group includes multiple messages. For ease of description, the format of the messages generated by the first device in this embodiment will be described below using the first message in a certain message group as an example:

[0128] As one possible implementation, if the first message is a transaction layer message, then the first message includes a transaction layer header (TA header), and the TA header includes the first sequence number (SN), first indication information, and second indication information of the first message. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the ordering requirements of the first message. The first SN can be an SSN. For example, if the transaction layer message is large and multiple transaction layers share a single transport layer, to prevent a message from a certain transaction layer from occupying the transport layer connection for a long time, the transaction layer can divide the message into multiple slices (e.g., a slice of 64KB) when sending the message to the transport layer. A transaction layer sends only one slice to the transport layer at a time, and each slice includes the corresponding SSN.

[0129] Specifically, in this implementation, after the transport layer of the first device receives the first message, it can determine the transmission order of the first message based on the first SN included in the TA header, and determine to transmit the first message to the transport layer of the second device within the TPG corresponding to the first message based on the first indication information, thereby achieving load balancing within the TPG. Additionally, it determines whether the first message needs to be transmitted in order based on the second indication information.

[0130] In this application, a message requiring ordered transmission (or a message with an ordered transmission requirement) refers to a message that can only be sent when its preceding message has been successfully received by the receiver, and / or when subsequent messages have not been successfully received by the receiver. A message not requiring ordered transmission (or a message without an ordered transmission requirement) refers to a message whose transmission does not need to consider the transmission and reception of other messages. Specifically, a message requiring ordered transmission with a preceding message can be called a preceding-order-ordered message; a message requiring ordered transmission with a following message can be called a following-order-ordered message; and a message requiring ordered transmission with both preceding and following messages can be called a pre- and post-order-ordered message.

[0131] Additionally, it should be noted that the messages requiring ordered transmission in this application can also be messages with other ordered transmission requirements. For example, a message can only be sent if at least one other message has not been successfully received by the receiving end. In other words, the transmission of messages requiring ordered transmission in this embodiment needs to take into account the transmission status of other messages. Specifically, the forms of messages requiring ordered transmission will not be listed here. For ease of description, the following description will use a preceding ordered message as an example.

[0132] As another possible implementation, if the first message is a transport layer message, then the first message includes a transaction layer header (TA header) and a transport layer header (TP header). The TA header includes the first message's first serial number (SN), first indication information, and second indication information. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the ordering requirement of the first message. The TP header contains the second SN of the first message.

[0133] In this implementation, message order preservation can be achieved at the transport layer.

[0134] For ease of description, the following text will use the first message as an example of a transaction layer message.

[0135] The message transmission method provided in this application is applied to a scenario where a first device and a second device transmit messages, and the first device generates multiple message groups. In this embodiment, the first device is a message sending device, and the second device is a message receiving device. For example, the first device can be the initiating node shown in Figure 2 above, and the second device can be the target node shown in Figure 2 above.

[0136] Specifically, establishing a Queue Pair (QP) connection between the first and second devices can be understood as establishing a one-to-one QP connection. This QP connection transmits messages through multiple paths (or links). As shown in Figure 2, the QP connection between the first and second devices can transmit messages through at least one TPG. Each TPG contains multiple TPs, and message load balancing is achieved among the at least one TP within a TPG. Messages from different TPs can be transmitted through different network paths, congestion control is implemented at the TP granularity, and retransmission is used to ensure reliable transmission in case of packet loss.

[0137] It should be understood that this application embodiment does not describe how a QP connection is established between the first device and the second device, or how messages can be transmitted through multiple paths. Please refer to descriptions of multi-path message transmission in current related technologies. This application embodiment mainly concerns how to achieve message order preservation while avoiding congestion in multi-path transmission scenarios.

[0138] Furthermore, after the first device generates a message, it can transmit multiple message groups containing messages through multiple paths according to the order preservation requirements. Therefore, the method flow shown in Figure 5 also includes:

[0139] S520, the first device sends a message to the second device.

[0140] Specifically, the first device transmits multiple message groups containing messages through multiple paths in accordance with the order preservation requirements.

[0141] In this embodiment, after the second device receives the message, it can process the message according to the message order requirements. Therefore, the method flow shown in Figure 5 further includes:

[0142] S530, the second device processes multiple messages.

[0143] The following will describe the processing method of the second device for messages with reference to specific embodiments, which will not be elaborated in this embodiment.

[0144] In this embodiment, the message ordering requirement can be implemented by the sending end to send messages in order or by the receiving end to receive messages in order. For ease of understanding, the following will describe the process of sending ordering (also known as source ordering mode) or receiving ordering (also known as destination ordering mode) in conjunction with specific embodiments.

[0145] Figure 6 illustrates a method for sending messages according to an embodiment of this application. This method enables the sending device to send ordered messages without blocking out-of-order messages.

[0146] Specifically, the sending device (i.e., the first device) stores a first mapping table and a sending queue. The first mapping table is used to indicate the messages that the receiving device (i.e., the second device) has received; for example, the first mapping table stores the identifiers of the messages that the second device has received. The sending queue includes messages to be sent, and can be applied to the multi-link message transmission scenario shown in Figure 2. The method includes the following steps:

[0147] S610, the first device receives the first message.

[0148] Specifically, the first device retrieves a first message from the transmission queue, the first message including a first SN and second indication information.

[0149] The first SN of the first message is used to indicate which number the first message is sent.

[0150] Optionally, the first message can be a transaction layer message, and the first SN is the segment sequence number (SSN). For example, the first device can assign an SSN to each transaction layer message (or WQE) it sends, and the SSN number increments with each message sent.

[0151] Optionally, the first message can be a transport layer message. The first device can split a large transaction WQE into TPWQEs, numbered TPMSN, and transmitted to the TP. The TP layer receives the TP WQEs formed after the TA split. For each TP WQE processed, each TP WQE is sent to a different TPG for transmission within the TP. After being balanced within the TPG, it is sent to a certain TP. For a certain TP, the data corresponding to the TP WQE is split into messages for transmission.

[0152] The second indication information is used to indicate that the first message is a message that needs to be sent in order, and the first message can be called an order-preserving message.

[0153] It should be understood that the ordered message involved in this embodiment refers to a message that needs to be sent in the correct order. For example, if the first message is a message with a sending sequence number of 3, and the first message needs to be sent in the correct order, then the first message can only be sent if the preceding message (i.e., the message with sending sequence numbers of 1 and 2) is successfully received by the receiving device.

[0154] In this embodiment, the first device acts as the sending device. To ensure ordered transmission of packets in a multi-link scenario, the first device constructs the relevant ordered data structure in the following manner:

[0155] Figure 7 is a schematic diagram of a data structure constructed by a transmitter according to an embodiment of this application. As shown in Figure 7, the context of the transmission queue (which can be called the work queue context WQC) saved by the transmitter device includes: a first index, a first valid flag, and a first queue index.

[0156] The first index (the bitmap_idx#1 field shown in Figure 7) is used to indicate the first mapping relationship described above. For example, the first index points to a Bitmap entity in the shared buffer (TX BitmapBuffer shown in Figure 7, which includes N Bitmap entities numbered IDX-0 to IDX-N). Optionally, the first index can be understood as the index of the order-preserving Bitmap used by the sending device.

[0157] The first valid flag (the `scr_order_wqe_vld` field shown in Figure 7) is used to indicate whether there are any ordered packets to be sent in the transmission queue. For example, the first valid flag indicates whether there are any ordered packets (which can be called WQEs) to be sent in the transmission queue. Each WQE contains a corresponding ordered flag (the `so_flag` field shown in Figure 7) and other fields (the `other filed` field in the WQE shown in Figure 7). Optionally, the first valid flag can be understood as the ordered WQE valid flag.

[0158] The first queue index (the `scr_order_wqe_idx` field shown in Figure 7) is used to buffer out-of-order messages that are not sent in time. For example, the first queue index indicates a WQE queue in the shared buffer (the `order WQEBuffer` shown in Figure 7, which includes N WQE queues numbered IDX-0 to IDX-N). Optionally, the first queue index can be understood as the out-of-order WQE queue index.

[0159] For example, the context of the send queue may also include other fields (such as the otherfield field in WQC as shown in Figure 7).

[0160] To facilitate understanding, the meaning of the different fields in Figure 7 is explained in detail below with reference to Table 2:

[0161] Table 2

[0162]

[0163]

[0164] As one possible implementation, the so_flag field in Table 2 includes two bits, such as: "00" indicating that the current message has no ordering requirement with other messages; "01" indicating that the current message has an ordering requirement with subsequent strongly ordered messages, and the strongly ordered message cannot be sent before the current message, which can be understood as the current message having a follow-up message; "10" indicating that the current message is a strongly ordered message; "00" is a reserved bit.

[0165] As another possible implementation, the so_flag field in Table 2 includes 1 bit, such as "0" indicating that the current message and other messages have no ordering requirements; "1" indicating that the current message has ordering requirements.

[0166] As described above, in this embodiment, the context of the sending queue constructed by the sending device includes a first index indicating the first mapping table and a first queue index of the first queue used to cache out-of-order messages that were not sent in time. Therefore, when creating the sending queue, the sending device can apply for a Bitmap entity based on the queue number of the sending queue to store the identifier of the message that the receiving end has successfully received, and can also apply for a WQE queue based on the queue number of the sending queue to store out-of-order messages that were not sent in time.

[0167] It should be understood that this embodiment does not limit how the sending device obtains the first mapping table and WQE queue. It can be obtained by randomly requesting from the cache according to the queue number, or it can obtain the first mapping table and WQE queue corresponding to the queue number from the cache by JFS according to the queue number.

[0168] Furthermore, as described above, the first valid flag included in the context of the transmission queue indicates whether there are any ordered WQEs to be transmitted in the transmission queue. Each WQE contains a corresponding ordering flag. Therefore, when the transmitting device creates a WQE, before writing the WQE that needs to be ordered into the transmission queue, the software sets the so_flag in the WQE so that when scheduling the ordered WQE during hardware logic processing, it can determine whether the ordered WQE needs to be transmitted in order based on the so_flag in the WQE.

[0169] Furthermore, in this embodiment, after the first device obtains the first message, it determines whether the first message can be sent immediately. The method flow shown in Figure 6 further includes:

[0170] S620, the first device determines whether to send the first message.

[0171] As can be seen from the above, the first message can be a preordered message, so the first message needs to be sent on the premise that the preorder message of the first message has been successfully received by the receiving device.

[0172] As one possible implementation, the first message is the first message sent by the first device, meaning that the first message has no preceding message. For example, the first SN of the first message is 1. In this implementation, the first device determines that it can directly send the first message.

[0173] As another possible implementation, all preceding messages of the first message have been successfully received by the receiving device. For example, the first message has a first SN of 3, and messages with SNs of 1 and 2 have been successfully received by the receiving device, and the first mapping table includes the identifiers of messages with SNs of 1 and 2. In this implementation, the first device determines that it can directly send the first message.

[0174] To facilitate understanding, a specific example will be used to illustrate how the first device determines that it can directly send the first message under this implementation method:

[0175] Example 1: The first device's transmission queue contains messages A, B, C, and D in sequence. Message C is the message that needs to be transmitted in order, including an order-keeping identifier (such as the second indication information included in the first message above). Messages A and B are the preceding messages of message C. After the first device retrieves message C from the transmission queue, it determines that message C needs to be transmitted in order, and since the first mapping table stores the TAACK corresponding to messages A and B, the first device determines that it can directly send message C.

[0176] As another possible implementation, at least one preceding message of the first message is not successfully received by the receiving device. For example, if the first message's first SN is 3, and messages with SNs of 1 and / or 2 are not successfully received by the receiving device, the first mapping table includes identifiers for messages with SNs of 1 and / or 2. In this implementation, the first device determines not to send the first message, buffers it in the WQE queue, and sends it only after the sending conditions are met. Furthermore, in this implementation, after buffering the first message, the first device can send subsequent messages that do not require order maintenance.

[0177] To facilitate understanding, a specific example will be used to illustrate how the first device determines whether to cache the first packet under this implementation method:

[0178] Example 2: The first device's transmission queue contains messages A, B, C, and D in sequence. Message C is the message that needs to be transmitted in order, including an order-keeping identifier (such as the second indication information included in the first message above). Messages A and B are the preceding messages of message C. After retrieving message C from the transmission queue, the first device determines that message C needs to be transmitted in order, and since the first mapping table does not store the TAACK corresponding to message A and / or B, the first device determines that message C needs to be buffered in the WQE queue, waiting for the transmission conditions to be met (i.e., the TAACKs corresponding to messages A and B have been received) before transmission. Furthermore, the first device sends message D, which does not require order-keeping, thus avoiding message C blocking the transmission of message D.

[0179] The above implementation methods detail the processing method of the first device for ordered messages. That is, in this embodiment, the first device determines whether the ordered WQE can be scheduled. If it cannot be scheduled, it sends subsequent non-ordered WQE messages, and simultaneously stores the ordered messages in the ordered WQE queue (e.g., src_order_wqe shown in Figure 6) and updates the first valid flag (e.g., src_order_wqe_vld flag shown in Figure 7) to indicate that there are ordered messages to be sent in the WQE queue.

[0180] Furthermore, the first device can update the Bitmap state in the WQEC based on the ACK returned by the second device, and the first device can determine whether the ordered packets in the WQE queue meet the scheduling requirements based on the maintained bitmap state and the order preservation valid flag.

[0181] For example, after the first device buffers the first message into the WQE queue, it receives an ACK from the second device. The ACK indicates that the second device has successfully received the second message, which is a preceding message of the first message. If the second device successfully receives the second message, it means that all preceding messages of the first message have been successfully received. After receiving the ACK, the first device updates the first mapping table, saves the identifier of the second message into the first mapping table, and determines that all preceding messages of the first message have been received based on the message identifier saved in the first mapping table. Thus, the sending condition of the first message has been met, and the first device can send the first message.

[0182] The message sending method shown in Figure 6 allows the sending device to construct an ordered data structure that can be stored in on-chip static random access memory (SRAM) or memory. While meeting the user's sending order requirements, it can avoid blocking subsequent out-of-order packets and fully utilize the bandwidth of the Internet. This ordered method can also be applied to the TP layer of the sending device for ordered data transmission.

[0183] This application also provides a method for receiving messages, through which the sending device can receive ordered messages. The method for receiving messages is described in detail below with reference to Figure 8.

[0184] Figure 8 illustrates a method for sending messages according to an embodiment of this application, applied to a second device. Specifically, the sending device (i.e., the second device) stores a second mapping table, which is used to indicate messages already received by the receiving device (i.e., the second device). This method can be applied to the multi-link message transmission scenario shown in Figure 2. The method includes the following steps:

[0185] S810, the second device receives the first message.

[0186] Specifically, the second device receives a first message from the first device. The first message includes a first SN and second indication information. The description of the first SN and the second indication information can be found in the embodiment shown in Figure 6, and will not be repeated here.

[0187] In this embodiment, the second device acts as the receiving device. To ensure that received packets are received in an ordered manner in a multi-link scenario, the second device constructs the relevant ordered data structure in the following way:

[0188] Figure 9 is a schematic diagram of a data structure constructed by a receiving end according to an embodiment of this application. As shown in Figure 9, the context of the receiving queue (which can be called the work queue context WQC) stored by the receiving end device includes: a second index, a second valid flag, and a second queue index.

[0189] The second index (the bitmap_idx#2 field shown in Figure 9) indicates the second mapping relationship described above. For example, the second index points to a Bitmap entity in the shared buffer (RX BitmapBuffer shown in Figure 9, which includes N Bitmap entities numbered IDX-0 to IDX-N). Each Bitmap entity includes a pre-order flag and a reach flag. Optionally, the second index can be understood as the index of the pre-order Bitmap used by the receiving device.

[0190] The second valid flag (the dst_order_wqe_vld field shown in Figure 9) is used to indicate whether the receive queue contains a received ordered WQE. Each WQE contains an index to the temporary storage management list of received data. For example, the second valid flag indicates whether the receive queue contains a received ordered message (which can be called an ordered WQE). Each ordered WQE contains a corresponding linked list flag (the link_idx field shown in Figure 9) and other fields (the otherfield field in the WQE shown in Figure 9). Optionally, the second valid flag can be understood as the ordered WQE valid flag.

[0191] The second queue index (the dst_order_wqe_idx field shown in Figure 9) is used to buffer out-of-order messages that are not processed in a timely manner. For example, the second queue index indicates a WQE queue in the shared buffer (the order WQEBuffer shown in Figure 9, which includes N WQE queues numbered IDX-0 to IDX-N). Optionally, the second queue index can be understood as the index of the out-of-order WQE queue.

[0192] For example, the context of the receive queue may also include other fields (such as the otherfield field in WQC as shown in Figure 9).

[0193] Specifically, after receiving the first message, the second device parses the header of the first message to obtain information about the first message, including but not limited to the following information: the PSN, TPMSN, SN, second indication information (e.g., order preservation so_flag), and receive queue number information of the first message.

[0194] Furthermore, the second device obtains the context information of the receive queue based on the header information of the first message, such as obtaining the context information of the receive queue based on the receive queue number. And the first device determines the processing method for the first message based on the context information of the receive queue. The method flow shown in Figure 8 further includes:

[0195] S820, the second device determines the processing method for the first message.

[0196] As one possible implementation, if the first message does not carry an order-keeping flag, the second device can process the first message out of order. For example, it can process the message data based on the order-keeping information in the context information of the receive queue and the parsed information. For instance, based on the memory address obtained by WQEC, it can write the corresponding message data into memory, synchronously update the second mapping table information, and trigger the order-keeping message writing judgment based on the updated second mapping table information.

[0197] As another possible implementation, the first message carries a sequence-keeping flag. If it carries the aforementioned second indication information, the second device can perform sequence-keeping processing on the first message. For example, based on the sequence-keeping information in WQEC and the parsed PSN, TPMSN, SN, and sequence-keeping so_flag, the message data is processed, the second sequence-keeping mapping table information is updated synchronously, the content of the first message is written to the temporary buffer space, the corresponding linked list state is updated, and the sequence-keeping message write judgment is triggered.

[0198] Specifically, the order-keeping message write judgment includes: based on the current state of the second mapping table, under the condition that the preceding message has arrived, reading the order-keeping message data in the order-keeping cache and writing it into memory, and updating the corresponding cache list state and the state of the second mapping table.

[0199] The message receiving method shown in Figure 8 does not necessarily require a reorder buffer when the sending device maintains order or when a single TP (Transmission Terminal) uses only one flow and one network path. The relevant data structures can be stored in on-chip SRAM or memory. This method can fully utilize the bandwidth of the interconnection network while ensuring the user fulfills the execution order requirements. The receiver-side order-maintaining method can be applied at the TP layer and TA layer, maintaining the bitmap state based on TPMSN and SN, respectively.

[0200] It should be understood that the specific examples shown in Figures 5 to 9 of the embodiments of this application are only for the purpose of helping those skilled in the art to better understand the embodiments of this application, and are not intended to limit the scope of the embodiments of this application. It should also be understood that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0201] It should also be understood that, unless otherwise specified or logically conflicting, the terminology and / or descriptions in different embodiments of this application are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. For example, the current page table management mechanism merges address and permission management, storing them in the same page table. However, in some application scenarios, a host may open the same address space to multiple users, but each user's permissions are different. Under the current page table mechanism, multiple copies of the page table can only be used to meet the needs of this scenario. From another perspective, if address translation and permission verification are managed separately, multiple users can share one address table but each user has their own permission table, which can well adapt to this scenario. Given the above-mentioned permission-address separation design, this application can play a significant role in the design of the permission table because permission management is more centralized than address management, and the representation of a range table is more suitable for permission management of large VA spaces.

[0202] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] The following describes in detail the message transmission apparatus provided in the embodiments of this application with reference to Figures 10 to 12. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, for content not described in detail, please refer to the method embodiments above. For the sake of brevity, some content will not be repeated.

[0204] This application embodiment can divide the transmitting or receiving device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation. The following description uses the division of functional modules according to each function as an example.

[0205] Figure 10 shows a schematic diagram of the structure of a message transmission device 1000 provided in an embodiment of this application.

[0206] In one example, the message transmission apparatus 1000 can be applied to a first device, and can be used to execute the message transmission method described above, such as the method shown in FIG5 or FIG6. The message transmission apparatus 1000 stores a first mapping table and a sending queue. The first mapping table includes identifiers of messages already received by the second device, and the sending queue includes messages to be sent. Specifically, the message transmission apparatus 1000 may include an acquisition unit 1010, a processing unit 1020, and a transceiver unit 1030.

[0207] The acquisition unit 1010 is used to acquire a first message from the transmission queue, wherein the first message is a pre-ordered message. The processing unit 1020 is used to determine whether to send the first message; if the identifiers of the preceding messages of the first message are all in the first mapping table, the transceiver unit 1030 is used to send the first message; if the identifiers of at least one preceding message of the first message are not in the first mapping table, the acquisition unit 1010 is used to acquire the non-pre-ordered messages following the first message, and the transceiver unit 1030 is used to send the non-pre-ordered messages following the first message.

[0208] As an example, referring to Figure 5, the processing unit 1020 can be used to execute S510, and the transceiver unit 1030 can be used to execute S520.

[0209] As an example, referring to Figure 6, the acquisition unit 1010 can be used to execute S610, and the processing unit 1020 can be used to execute S620.

[0210] It should be noted that the apparatus described in FIG10 can also be used to perform the method steps involved in the aforementioned variations of the embodiments shown in the figures, which will not be repeated here.

[0211] In another example, the message transmission apparatus 1000 can be applied to a second device, and the message transmission apparatus 1000 can be used to perform the message transmission method described above, for example, to perform the method shown in FIG5 or FIG8. The message transmission apparatus 1000 stores a second mapping table, which includes identifiers of messages that have been received by the second device.

[0212] The transceiver unit 1030 is used to receive a first message from the first device, save the identifier of the first message to the second mapping table, and update the second mapping table. The first message is a pre-order message. When the identifiers of the preceding messages of the first message are all in the second mapping table, the processing unit 1020 is used to process the first message and the preceding messages of the first message. When the identifier of at least one preceding message of the second message is not in the second mapping table, the processing unit 1020 is used to buffer the first message. The transceiver unit 1030 is used to receive non-pre-order messages after the first message.

[0213] As an example, referring to Figure 5, the processing unit 1020 can be used to execute S530, and the transceiver unit 1030 can be used to execute S520.

[0214] As an example, referring to Figure 8, the transceiver unit 1030 can be used to execute S810, and the processing unit 1020 can be used to execute S820.

[0215] It should be noted that the apparatus described in FIG10 can also be used to perform the method steps involved in the aforementioned variations of the embodiments shown in the figures, which will not be repeated here.

[0216] This application embodiment also provides a chip system 1100, as shown in FIG11. The chip system 1100 includes at least one processor and at least one interface circuit. As an example, when the chip system 1100 includes a processor and an interface circuit, the processor may be the processor 1110 shown in the solid box in FIG11 (or the processor 1110 shown in the dashed box), and the interface circuit may be the interface circuit 1120 shown in the solid box in FIG11 (or the interface circuit 1120 shown in the dashed box).

[0217] When the chip system 1100 includes two processors and two interface circuits, the two processors include processor 1110 shown in the solid box and processor 1110 shown in the dashed box in FIG11, and the two interface circuits include interface circuit 1120 shown in the solid box and interface circuit 1120 shown in the dashed box in FIG11. This is not limited. Processor 1110 and interface circuit 1120 can be interconnected via lines. For example, interface circuit 1120 can be used to receive signals (e.g., instructions stored in memory). As another example, interface circuit 1120 can be used to send signals to other devices (e.g., processor 1110).

[0218] For example, interface circuit 1120 can read instructions stored in memory and send those instructions to processor 1110. When the instructions are executed by processor 1110, the message transmission device can perform the steps in the above embodiments. Of course, the chip system 1100 may also include other discrete devices, and this application embodiment does not specifically limit this.

[0219] Another embodiment of this application provides a computer-readable storage medium storing instructions that, when executed on a message-transmitting apparatus, perform the various steps of the message-transmitting apparatus in the method flow shown in the above-described method embodiments. In some embodiments, the disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or encoded on other non-transitory media or articles of art.

[0220] Figure 12 schematically illustrates a conceptual partial view of a computer program product provided in an embodiment of this application, the computer program product including a computer program for executing computer processes on a computer device.

[0221] In one embodiment, the computer program product is provided using a signal carrying medium 1200. The signal carrying medium 1200 may include one or more program instructions that, when executed by one or more processors, provide the functions or parts thereof described above with reference to Figures 5, 6, and 8. Therefore, for example, one or more features of S510–S530 in Figure 5 may be embodied by one or more instructions associated with the signal carrying medium 1200. Furthermore, example instructions are also described in the program instructions of Figure 12.

[0222] In some examples, the signal carrying medium 1200 may include a computer-readable medium 1201, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, a read-only memory (ROM), or a random access memory (RAM), etc.

[0223] In some implementations, the signal carrying medium 1200 may include a computer recordable medium 1202, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on.

[0224] In some implementations, the signal-bearing medium 1200 may include a communication medium 1203, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). The signal-bearing medium 1200 may be transmitted by a wireless communication medium 1203 (e.g., a wireless communication medium conforming to the IEEE 1502.11 standard or other transmission protocols). One or more program instructions may be, for example, computer-executable instructions or logical implementation instructions.

[0225] In some examples, such as the apparatus for transmitting messages in Figure 5, the apparatus may be configured to provide various operations, functions, or actions in response to one or more program instructions in a computer-readable medium 1201, a computer-recordable medium 1202, and / or a communication medium 1203.

[0226] It should be understood that the arrangements described herein are for illustrative purposes only. Therefore, those skilled in the art will understand that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements may be omitted depending on the desired outcome. Furthermore, many of the described elements are functional entities that can be implemented as discrete or distributed components, or in any suitable combination and location with other components.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, it can be implemented, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When executed on a computer and when the computer execution instructions are executed, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0228] Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access, or it can include one or more data storage devices such as servers or data centers that can be integrated with media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0229] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for message transmission, characterized in that, The method is applied to a first device, which transmits messages with a second device via at least one Transmission Link Group (TPG). The first device stores a first mapping table and a sending queue. The first mapping table includes identifiers of messages that have been received by the second device, and the sending queue includes messages to be sent. The method includes: when the identifier of at least one preceding message of the first message in the sending queue is not in the first mapping table, the first device obtains and sends non-order-keeping messages following the first message, wherein the first message is an order-keeping message.

2. The method according to claim 1, characterized in that, When the identifiers of the preceding messages of the first message are all in the first mapping table, the first device sends the first message.

3. The method according to claim 1 or 2, characterized in that, The first message is a transaction layer message, and the first message includes a transaction layer header (TA header). The TA header includes a first sequence number (SN), a first indication information, and a second indication information for the first message. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the order preservation requirement of the first message.

4. The method according to claim 3, characterized in that, Sending the first message by the first device includes: the transaction layer of the first device sending the first message to the transport layer of the first device; and the transport layer of the first device sending the first message to the transport layer of the second device.

5. The method according to claim 3, characterized in that, After the first device sends the first message, the method further includes: the first device saving the first SN to the first mapping table.

6. The method according to claim 1, characterized in that, If the identifier of at least one preceding message of the first message is not in the first mapping table, the method further includes: the first device caching the first message in the cache of the first device.

7. The method according to claim 6, characterized in that, The method further includes: the first device receiving confirmation information from the second device, the confirmation information indicating that the second message has been received by the second device; the first device saving the identifier of the second message to the first mapping table and updating the first mapping table; and the first device determining whether to send the first message based on the updated first mapping table.

8. The method according to claim 1, characterized in that, The first device acquires and sends a non-orderly message following the first message, including: the first device acquires a third message following the first message from the sending queue; the first device determines that the third message is a non-orderly message based on the ordering requirement carried in the transaction layer header (TA) of the third message; and the first device sends the third message.

9. The method according to claim 1, characterized in that, The context of the sending queue includes the following information: a first index, a first valid flag, and a first queue index. The first index is used to indicate the first mapping table, the first valid flag is used to indicate whether there are any ordered messages to be sent in the sending queue, the first queue index is used to indicate the first queue, and the first queue is used to buffer unsent ordered messages.

10. The method according to claim 1, characterized in that, The method further includes setting second indication information when the first device creates an operation unit (WQE) that includes a transmission queue of the first message.

11. The method according to claim 1, characterized in that, The first device establishes a queue pair (QP) connection with the second device, and the QP connection transmits messages through at least one transmission link group (TPG).

12. A message transmission apparatus, characterized in that, The device and the second device transmit messages via at least one Transmission Link Group (TPG). The device stores a first mapping table and a sending queue. The first mapping table includes identifiers of messages that have been received by the second device. The sending queue includes messages to be sent. The device includes a processing module and a sending module. The processing module is used to obtain non-order-preserving messages following the first message when the identifier of at least one preceding message of the first message in the sending queue is not in the first mapping table, wherein the first message is an order-preserving message. The sending module is also used to send the non-order-preserving messages.

13. The apparatus according to claim 12, characterized in that, When the identifiers of the preceding messages of the first message are all in the first mapping table, the sending module is used to send the first message.

14. The apparatus according to claim 12 or 13, characterized in that, The first message is a transaction layer message, which includes a transaction layer header (TA). The TA header contains a first sequence number (SN), first indication information, and second indication information. The first indication information is used to indicate the TPG corresponding to the first message, and the second indication information is used to indicate the order preservation requirement of the first message.

15. The apparatus according to claim 14, characterized in that, The device includes a transaction layer and a transport layer. The sending module includes a sending module of the transaction layer and a sending module of the transport layer. The sending module sends the first message, including: the sending module of the transaction layer sending the first message to the transport layer; and the sending module of the transport layer sending the first message to the transport layer of the second device.

16. The apparatus according to claim 15, characterized in that, After the sending module sends the first message, the processing module is further configured to save the first SN to the first mapping table.

17. The apparatus according to claim 12, characterized in that, When the identifier of at least one preceding message of the first message is not in the first mapping table, the processing module is further configured to cache the first message in the cache of the device.

18. The apparatus according to claim 17, characterized in that, The device further includes: a receiving module, configured to receive confirmation information from the second device, the confirmation information indicating that the second message has been received by the second device; the processing module, configured to save the identifier of the second message to the first mapping table and update the first mapping table; the processing module, configured to determine whether to send the first message based on the updated first mapping table.

19. The apparatus according to claim 12, characterized in that, The sending module obtains the third message following the first message from the sending queue; the processing module determines that the third message is a non-order-preserving message based on the order-preserving requirement carried in the transaction layer header (TA) of the third message. The sending module sends the third message.

20. The apparatus according to claim 12, characterized in that, The context of the sending queue includes the following information: a first index, a first valid flag, and a first queue index. The first index is used to indicate the first mapping table, the first valid flag is used to indicate whether there are any ordered messages to be sent in the sending queue, the first queue index is used to indicate the first queue, and the first queue is used to buffer unsent ordered messages.

21. The apparatus according to claim 12, characterized in that, The processing module is further configured to set second indication information when creating an operation unit (WQE) that includes the first message's transmission queue.

22. The apparatus according to claim 12, characterized in that, The device establishes a queue pair (QP) connection with the second device, and the QP connection transmits messages through at least one transmission link group (TPG).

23. A message transmission apparatus, characterized in that, include: A processor for reading instructions stored in a memory, which, when executed, cause the message transmission apparatus to perform the method of any one of claims 1 to 11.

24. A message transmission apparatus, characterized in that, Includes units for performing the method as described in any one of claims 1 to 11.

25. A chip, characterized in that, include: At least one processing core is configured to perform the method as described in any one of claims 1 to 11.

26. A computer device, characterized in that, include: The chip as described in claim 25.

27. A computer program product, characterized in that, The computer program product includes computer program code, and when the computer program code is run on a computer, the method of any one of claims 1 to 11 is performed.

28. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on a computer device, causes a processing module in the computer device to perform the method as described in any one of claims 1 to 11.

29. A message transmission system, characterized in that, It includes a first device for performing the method as described in any one of claims 1 to 11 and a second device connected to the first device.

Images