Data transmission method, apparatus, device, and storage medium

Abstract

The application relates to a data transmission method, device, equipment and storage medium. The method comprises the following steps: acquiring data identifiers of data units in a data unit sequence, the data identifiers being used for representing transmission sequences; determining data writing positions based on the data identifiers; writing metadata corresponding to the data units in the data unit sequence into a sending queue according to the data writing positions; and sending a processing notification to a network card after completing the writing of the metadata of all the data units, so that the network card executes the data units corresponding to the metadata in the sending queue to a target node based on the data identifiers after receiving the processing notification. The method can improve data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data transmission method, apparatus, device, and storage medium. Background Technology

[0002] With the development of distributed large model training and inference technology, high-speed communication technology based on RDMA (Remote Direct Memory Access) has emerged. This technology transmits data between the local and remote devices through the work queue element (WQE) in the queue pair (QP), which can achieve low latency and high bandwidth data exchange.

[0003] In traditional technology, queue elements in the sending queue are usually written in the order of computation, and each queue element needs to be notified to the network card for processing immediately after being written into the sending queue. In other words, these notifications sent to the network card are sent serially, which makes it impossible to efficiently transmit order-sensitive data. Summary of the Invention

[0004] Therefore, it is necessary to provide a data transmission method, apparatus, device, and storage medium that can improve data transmission efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data transmission method. The method includes:

[0006] Obtain the data identifier used to characterize the transmission order of each data unit in the data unit sequence;

[0007] The data writing location is determined based on the data identifier;

[0008] According to the data writing position, the metadata corresponding to the data unit in the data unit sequence is written into the sending queue;

[0009] After writing the metadata of all the data units, a processing notification is sent to the network interface card (NIC) so that the NIC, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

[0010] Secondly, this application also provides a data transmission apparatus. The apparatus includes:

[0011] The data identifier acquisition module is used to acquire the data identifier of each data unit in the data unit sequence, which is used to represent the transmission order;

[0012] A write location determination module is used to determine the data write location based on the data identifier;

[0013] The metadata writing module is used to write the metadata corresponding to the data unit in the data unit sequence into the sending queue according to the data writing position;

[0014] The data execution module is used to send a processing notification to the network interface card (NIC) after writing the metadata of all the data units, so that the NIC, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the methods described above.

[0016] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0017] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0018] The aforementioned data transmission method, apparatus, computer equipment, storage medium, and computer program product acquire data identifiers representing the transmission order of each data unit in the data unit sequence, determine the data writing position based on the data identifiers, and write the metadata corresponding to the data units in the data unit sequence into the transmission queue according to the data writing positions. After completing the writing of the metadata of all data units, a processing notification is sent to the network card, so that after receiving the processing notification, the network card executes the data units corresponding to the metadata in the transmission queue to the target node based on the data identifiers. This pre-establishes an ordered arrangement relationship of data units in the transmission queue according to the data identifiers, enabling the network card to process each data unit in the order of transmission priority. At the same time, the network card is notified to process the data units after all metadata is written, reducing the number of notifications and the latency overhead caused by serial triggering, avoiding multiple waiting and processing blockages, thereby improving the data transmission efficiency in sequence-sensitive scenarios and enhancing the parallel processing capability and overall performance of the system. Attached Figure Description

[0019] Figure 1 This is an application environment diagram of a data transmission method in one embodiment;

[0020] Figure 2 This is a flowchart illustrating a data transmission method in one embodiment;

[0021] Figure 3This is a schematic diagram of the queue element writing process in one embodiment;

[0022] Figure 4 This is a schematic diagram of the queue element writing process in another embodiment;

[0023] Figure 5 This is a schematic diagram illustrating the principle of a data transmission method in one embodiment;

[0024] Figure 6 This is a schematic diagram illustrating the principle of a data transmission method in another embodiment;

[0025] Figure 7 This is a schematic diagram illustrating the principle of sending queue allocation in one embodiment;

[0026] Figure 8 This is a flowchart illustrating the data transmission method in another embodiment;

[0027] Figure 9 This is a schematic diagram illustrating the principle of a data transmission method in another embodiment;

[0028] Figure 10 A timing diagram of the data transmission method in another embodiment;

[0029] Figure 11 This is a schematic diagram illustrating the effect of a data transmission method in one embodiment;

[0030] Figure 12 This is a schematic diagram illustrating the effect of the data transmission method in another embodiment;

[0031] Figure 13 This is a structural block diagram of a data transmission device in one embodiment;

[0032] Figure 14 This is a structural block diagram of a data transmission device in another embodiment;

[0033] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] The data transmission method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, computer device 102 communicates with a target node (such as node 104 or node 106) via a network. The above data transmission method can be executed by computer device 102. Taking node 104 as the target node, the above data transmission method is explained as follows: Computer device 102 obtains the data identifier of each data unit in the data unit sequence, which is used to characterize the transmission order. Based on the data identifier, it determines the data writing position. According to the data writing position, it writes the metadata corresponding to the data unit in the data unit sequence into the sending queue. After completing the writing of the metadata of all data units, it sends a processing notification to the network card, so that after receiving the processing notification, the network card executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

[0036] In this system, computer device 102 can be a client device accessing the distributed system. This client device can be a physical server, virtual machine, container instance, or other computing node with data read / write capabilities. It can also be a terminal, which can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Computer device 102 can also be a node in the distributed system. The target node can be a node in the distributed system. It is understood that the distributed system can include multiple nodes. A node can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0037] In one embodiment, such as Figure 2 As shown, a data transmission method is provided, which is applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0038] S202, Obtain the data identifier used to characterize the transmission order of each data unit in the data unit sequence.

[0039] Here, a data unit sequence refers to an ordered set of multiple data units to be transmitted arranged according to preset rules (such as transmission priority or calculation order). It can be understood that multiple data units in this data unit sequence are data units that need to be transmitted to the target node; it refers to the receiving end device or processing unit for data transmission, such as a computing node in a distributed system used to receive and process data units; in a distributed system, the target node can specifically be a processing unit deployed on different computing devices, such as expert nodes, GPU devices, computing instances, or other functional nodes that can process data units in a model.

[0040] A data unit refers to a data fragment that serves as the basic processing and transmission object during data transmission. In a distributed system, the data to be transmitted can be stored or generated by any node in the distributed system. For example, when training or inferring a large artificial intelligence model in a distributed system, the data to be transmitted can be intermediate result data, feature vectors, etc., generated during the model calculation process. This data to be transmitted can be split or divided into multiple independent data fragments, thus obtaining multiple data units to be transmitted. Specifically, these data units can be mapped or corresponded to tokens as the basic processing units for distributed computing and data transmission.

[0041] Transmission order refers to the pre-defined order in which multiple data units are transmitted to the target node. It is used to indicate the order in which each data unit is processed and transmitted by the network card in the transmission queue, so as to ensure the correctness of data processing in order-sensitive scenarios. For example, when training or inferring large artificial intelligence models in a distributed system, different data units may correspond to different processing priorities or computational dependencies. By pre-setting the transmission order, data units with higher priority or those on which subsequent calculations depend can be transmitted to the target node first, thereby avoiding computational delays or incorrect results caused by inconsistent data arrival order.

[0042] Data identifiers are information used to characterize the position of a corresponding data unit in the transmission order. They can be sequence numbers, index values, or other identifiers that can uniquely determine their relative order in the data unit sequence. For example, when there are multiple data units, the data identifiers corresponding to each data unit can be 1, 2, 3, ..., N. The numerical value of the data identifier is used to indicate the transmission order of the corresponding data units. Data units corresponding to data identifiers with smaller numerical values ​​are transmitted to the target node first.

[0043] Specifically, the computer device can acquire a sequence of data units for a target node and acquire a data identifier pre-configured for each data unit in the sequence to characterize the transmission order, wherein the data identifier is used to characterize the relative transmission order of the data unit in the sequence.

[0044] In one embodiment, the data identifier can be generated based on the transmission priority of the data unit or a preset sorting rule, and assigned according to the sorting position of the data unit in the data unit sequence.

[0045] S204, determine the data writing location based on the data identifier.

[0046] Here, the data write position refers to the write position of the metadata corresponding to each data unit in the transmission queue, which is used to characterize the queue slot of the corresponding data unit in the transmission queue. The metadata of the data unit refers to the data set used to describe the transmission information of the corresponding data unit. The transmission information includes, but is not limited to, the storage address of the data unit in the transmission buffer, the data length, and the write position parameters of the target node, which is used to guide the network card to transmit data to the corresponding data unit. The transmission queue refers to the queue structure associated with the network card, which is used to store the queue elements to be transmitted. The queue elements are composed of metadata. The network card performs the corresponding data transmission operation based on the queue elements in the transmission queue. The queue slot of the transmission queue refers to the basic storage unit in the transmission queue used to store a single queue element. Each queue slot is used to store the metadata corresponding to a data unit and is distinguished by a position index.

[0047] It is understood that data transmission in this embodiment mainly adopts Remote Direct Memory Access (RDMA). In this mode, the network card can directly read the data of the corresponding data unit from the transmit buffer according to the queue element in the transmit queue, and write the data unit into the target memory area of ​​the target node without the participation of the central processing unit, thereby reducing data transmission latency and improving transmission efficiency.

[0048] Specifically, after the computer device obtains the data identifier of each data unit, it maps any data unit to the corresponding data write position based on a pre-set mapping rule. The mapping rule is used to combine the data identifier with the starting write position parameter of the sending queue to determine the queue slot of the data unit in the sending queue.

[0049] S206, write the metadata corresponding to the data unit in the data unit sequence into the sending queue according to the data writing position.

[0050] Specifically, the computer device can obtain the data writing position corresponding to each data unit and the metadata corresponding to each data unit; for any data unit, the computer device writes the metadata corresponding to the data unit as a queue element into the corresponding queue slot in the sending queue according to the data writing position, so as to complete the construction of the queue element.

[0051] For example, the data units to be transmitted corresponding to target node A are data unit 1, data unit 2, and data unit 3, and the sending queue corresponding to target node A is sending queue A. The computer device can determine the data writing position in sending queue A according to the data identifiers corresponding to data unit 1, data unit 2, and data unit 3, for example, corresponding to the k-th, k+1-th, and k+2-th queue slots in sending queue A, respectively. Subsequently, the computer device writes the metadata corresponding to data unit 1, data unit 2, and data unit 3 into the corresponding queue slots in sending queue A, thereby constructing the corresponding queue element sequence in sending queue A.

[0052] In one embodiment, the process of a computer device writing metadata corresponding to a data unit into a sending queue includes: determining the operation type corresponding to each data unit based on at least one of the transmission purpose of the data unit and the memory operation requirements of the target node, constructing a queue element based on the metadata corresponding to the data unit and the operation type, and writing the constructed queue element into the corresponding queue slot in the sending queue.

[0053] The transmission purpose of a data unit refers to the purpose or function of the data unit being transmitted from the sending end to the target node. For example, it may be used to transmit model calculation results, token data, feature vectors, or to control the state of the target node (such as counter updates or trigger operations).

[0054] The target node memory operation requirements refer to the processing method or requirements of the receiving end for the data unit in memory. For example, the receiving end needs to write data to a specified memory address, perform atomic addition operations to update the counter, or modify the flag bit to trigger subsequent calculations.

[0055] The operation type refers to the RDMA operation type specified in the queue element, which is used to guide the network card to perform the corresponding data transmission or atomic operation, such as Write or FetchAdd. Write is used to indicate that the data unit is completely written to the memory of the target node, and FetchAdd performs an atomic addition operation on the memory location of the target node to update the counter or status flag.

[0056] Specifically, the computer device can first obtain the transmission purpose and target node memory operation requirements of each data unit. The transmission purpose indicates whether the data unit needs to transmit actual data or is used for synchronization, counting, or triggering operations. The target node memory operation requirements indicate the type of operation that the receiving end should perform on the data unit. Subsequently, the computer device determines the operation type corresponding to each data unit based on at least one of this information. For example, it marks data that needs to be transmitted as Write and data used to update counters or status as FetchAdd. After determining the operation type, the computer device combines the metadata of each data unit (including local storage address, data length, target memory address, etc.) to construct a queue element (WQE). Finally, the computer device writes the constructed queue element into the corresponding queue slot in the transmit queue so that the network interface card can perform the corresponding write or atomic operation according to the operation type of the queue element after receiving the processing notification.

[0057] S208 After completing the writing of metadata for all data units, a processing notification is sent to the network interface card (NIC) so that the NIC, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

[0058] Specifically, writing metadata for all data units means that for all data units to be transmitted in the current batch, the corresponding metadata has been written as queue elements into the corresponding queue slots in the sending queue, and the metadata has been made visible to the network card through a memory barrier operation. For example, if the target node has 10 data units to be transmitted in the current batch, the computer device will write the metadata corresponding to each of the 10 data units as 10 queue elements into 10 consecutive or predetermined queue slots in the sending queue. After writing the above 10 queue elements, each target thread will execute a memory barrier operation to ensure that all 10 queue elements have been written to memory and are visible to the network card. At this point, it can be considered that the writing of metadata for all data units has been completed.

[0059] The processing notification is a control signal sent by the network card to trigger the network card to process queue elements in the transmission queue. The processing notification can be implemented by updating the doorbell register or writing the doorbell record, and is used to instruct the network card to start reading and processing queue elements in the transmission queue.

[0060] A network interface card (NIC) is a network interface device used to perform data transmission operations. Based on the transmission information described by the queue elements in the transmission queue, the NIC can directly read data units from the transmission buffer and write them to the target memory area of ​​the target node via the network. Specifically, the NIC here can refer to the NIC of the data sending end, i.e., the network interface device used to read data units from the sending end and initiate data transmission. In the corresponding data receiving process, the target node side can also include a NIC of the data receiving end, used to receive data units transmitted over the network and execute the data units in the target node's receive buffer or target memory area for subsequent processing.

[0061] Data unit execution to the target node refers to the process where, after the network interface card (NIC) receives the processing notification and performs the corresponding operation according to the queue element, it completes the operation in the target node's memory according to the metadata information in the queue element for each data unit described in the sending queue. This includes, but is not limited to, write operations or atomic operations (such as FetchAdd). A write operation refers to transmitting the complete content of the data unit from the sending buffer and writing it to the memory location specified by the target node. An atomic operation (such as FetchAdd) refers to performing atomic addition or other atomic operations on a counter or status variable at a specified address in the target node's memory without transmitting the data unit itself.

[0062] Specifically, after the computer device completes the writing of metadata for all data units into the transmission queue, it can send a processing notification to the network interface card (NIC) to trigger the NIC to process the queue elements in the transmission queue. Before sending the processing notification, the computer device can perform a memory barrier operation to ensure that the metadata written into the transmission queue is visible to the NIC. Subsequently, the computer device sends the processing notification by writing notification information to the NIC's doorbell register or updating the doorbell record to instruct the NIC to start reading the queue elements in the transmission queue. After receiving the processing notification, the NIC reads each queue element from the transmission queue in sequence and, based on the transmission information described in the queue element, reads the data of the corresponding data unit from the transmission buffer, and directly executes the data unit to the target memory area of ​​the target node via the network. The NIC can process each queue element sequentially based on the transmission order represented by the data identifier, thereby ensuring that the data units are executed to the target node in a preset order.

[0063] In one embodiment, after receiving a processing notification, the network interface card (NIC) reads each queue element sequentially from the transmission queue and executes the corresponding operation according to the operation type in the queue element. When the operation type corresponding to a queue element is Write, the NIC reads the corresponding data unit from the transmission buffer of the sending end according to the metadata recorded in the queue element, and writes the data unit completely to the specified memory address of the target node, while ensuring the order and integrity of the data, so that the target node can receive data in the transmission order represented by the queue element. When the operation type corresponding to a queue element is FetchAdd, the NIC reads the original value from the memory address specified by the target node according to the metadata recorded in the queue element, performs atomic addition on the operand specified in the queue element to update the counter or status variable in the target memory, and returns the original value (if necessary) to achieve synchronous update of the target node's status or count. This operation guarantees atomicity and order, so that the target node can correctly record the number of data units that have arrived or the status change, and execute in the same order as the operations of other queue elements, satisfying the requirements of order-sensitive communication.

[0064] In one embodiment, a computer device can call a queue element write interface to perform the following steps: obtaining data identifiers for each data unit in the data unit sequence to characterize the transmission order; determining the data write position based on the data identifiers; and writing the metadata corresponding to the data units in the data unit sequence into the transmission queue according to the data write position. In the above process, the queue element write interface is used to construct the metadata into queue elements and write them into the transmission queue without triggering network interface card (NIC) processing. After the metadata of all data units is written, a notification interface is called to send a processing notification to the NIC, which triggers the NIC to perform unified processing on the queue elements in the transmission queue, thereby realizing the batch transmission of data units.

[0065] The queue element write interface refers to the interface called by the computer device to construct the metadata corresponding to the data unit into a queue element and write it into the transmission queue. This interface only performs the queue element write operation and does not trigger the network card to process the queue elements in the transmission queue.

[0066] The notification interface is an interface called by a computer device to send processing notifications to the network card. The notification interface is used to trigger the network card to start reading and processing queue elements in the transmission queue, thereby executing the corresponding data transmission operation.

[0067] It should be noted that in the application embodiment, the queue element writing interface and the notification interface are set independently. By decoupling the queue element writing operation from the network card processing trigger operation, the network card does not need to be notified multiple times during the writing phase, thereby reducing the number of notifications and reducing data delivery latency.

[0068] Combination Figure 3 and Figure 4 The solution described in this application is as follows: In the diagram, `write` refers to a queue element of type `write` operation, `FetchAdd` refers to a queue element of type `FetchAdd`, and `Notify` refers to the processing notification sent by the network interface card (NIC), used to trigger the NIC to start processing queue elements in the transmission queue, such as... Figure 3 The diagram illustrates the queue element writing process in the traditional scheme. WQE (queue element) writing and network interface card (NIC) notification are handled by a unified interface. When the sending end repeatedly writes queue elements of type Write and FetchAdd, or in other words, when the sending end repeatedly initiates Write and FetchAdd requests, a Notify operation may be triggered after each queue element is written. This results in each queue element requiring multiple notifications to the NIC, thus increasing the total time spent sending requests. Figure 4 The diagram shown illustrates the queue element writing process of this scheme. The WQE writing and network card notification steps are decoupled: the writing operation of all queue elements no longer triggers network card notification immediately. Instead, after a batch of WQE writes is completed, all queue elements are notified to the network card for processing at once through a single Notify operation. This scheme significantly reduces the number of times the network card is notified, thereby reducing the total time spent on request delivery and improving communication efficiency.

[0069] In the aforementioned data transmission method, data identifiers representing the transmission order of each data unit in the data unit sequence are obtained. Based on the data identifiers, the data writing position is determined. According to the data writing position, the metadata corresponding to the data units in the data unit sequence is written into the transmission queue. After the metadata of all data units is written, a processing notification is sent to the network card. Upon receiving the processing notification, the network card executes the data units corresponding to the metadata in the transmission queue to the target node based on the data identifiers. This pre-establishes an ordered arrangement of data units in the transmission queue according to the data identifiers, enabling the network card to process each data unit according to the transmission priority. Simultaneously, the network card is notified to process the data units after all metadata is written, reducing the number of notifications and the latency overhead caused by serial triggering. This avoids multiple waiting and processing blockages, thereby improving the data transmission efficiency in sequence-sensitive scenarios and enhancing the parallel processing capability and overall performance of the system.

[0070] In one embodiment, the above data transmission method further includes the following process: obtaining at least two data units to be transmitted for the target node; obtaining the transmission priority corresponding to each data unit; sorting the at least two data units according to the transmission priority to obtain a data unit sequence; the process by which the computer device obtains the data identifier of each data unit in the data unit sequence to characterize the transmission order specifically includes the following steps: determining the data identifier of each data unit according to the sorting position of each data unit in the data unit sequence.

[0071] Transmission priority refers to a parameter used to characterize the relative importance or urgency of each data unit during data transmission, and is used to determine the transmission order among multiple data units. Transmission priority can be set based on the dependencies of data units in the computation process, the urgency of processing, or business strategies. For example, in the training or inference process of large artificial intelligence models, data units that are strongly related to subsequent computations or are scheduled first can be given higher transmission priority so that they can be transmitted to the target node first.

[0072] Specifically, the computer device can acquire multiple data units to be transmitted for the target node, and obtain the transmission priority set in advance for each data unit. The multiple data units are sorted according to the transmission priority to obtain a data unit sequence. The sorting can be arranged from high to low or from low to high according to the transmission priority. After obtaining the data unit sequence, the computer device can assign a corresponding data identifier to each data unit according to the sorting position of each data unit in the data unit sequence to represent the relative transmission order of the data unit in the data unit sequence.

[0073] In one embodiment, the transmission priority can be represented in numerical form, with different values ​​corresponding to different priority levels. The computer device can sort the data units according to the values ​​to generate a data unit sequence.

[0074] In one embodiment, the data identifier can be a sequence number or index value that corresponds one-to-one with the sorting position. For example, the first data unit corresponds to data identifier 1, the second data unit corresponds to data identifier 2, and so on, thereby realizing the identification and management of the data unit transmission order.

[0075] In the above embodiments, the computer device acquires at least two data units to be transmitted for the target node, obtains the transmission priority corresponding to each data unit, sorts the at least two data units according to the transmission priority, obtains a data unit sequence, determines the data identifier of each data unit according to the sorting position of each data unit in the data unit sequence, thereby establishing a clear transmission order for each data unit, and further determines the writing position of each data unit in the transmission queue based on the data identifier, so that the data units can be written into the transmission queue in a preset order and processed by the network card in order, avoiding order disorder in the data transmission process, and improving the data transmission efficiency and processing correctness in order-sensitive scenarios.

[0076] In one embodiment, the process by which a computer device acquires at least two data units to be transmitted for a target node specifically includes the following steps: acquiring at least two raw data units to be transmitted for the target node; and performing compression and quantization processing on the at least two raw data units respectively to obtain at least two data units to be transmitted.

[0077] The raw data unit refers to a data unit that has not been compressed or quantized before data transmission. It can be the raw data fragment generated or stored by each computing node in a distributed system, such as intermediate result data, feature vectors, or raw data representations corresponding to each processing object generated during the training or inference of a large artificial intelligence model.

[0078] Compression quantization refers to the process of compressing or reducing the precision of raw data units. By reducing the precision of data representation or reducing data redundancy, the data volume is reduced. For example, high-precision data is converted into low-precision representation or data is encoded and compressed, thereby reducing the bandwidth required for data transmission and improving data transmission efficiency.

[0079] Specifically, the computer device can acquire at least two raw data units for the target node. The raw data units can be derived from local calculation results or data to be processed stored in memory. Subsequently, the computer device performs compression and quantization processing on the at least two raw data units to obtain at least two data units to be transmitted. The compression and quantization processing can include data compression or precision reduction processing on the raw data units to reduce the data volume and reduce transmission bandwidth overhead.

[0080] In one embodiment, a computer device can quantize and map the numerical data in the original data unit, convert the high-precision value into a low-precision representation, and compress the data in combination with preset encoding rules, thereby generating a data unit suitable for transmission.

[0081] In the above embodiments, the computer device obtains at least two raw data units to be transmitted for the target node, performs compression and quantization processing on the at least two raw data units respectively, and obtains at least two data units to be transmitted, thereby reducing the size of the data to be transmitted, reducing the bandwidth overhead required for data transmission, and reducing network transmission latency. This provides a basis for writing the metadata corresponding to the data unit into the sending queue and for the network card to efficiently execute data transmission, thereby improving the overall data transmission efficiency.

[0082] In one embodiment, the data transmission method further includes the following steps: sending at least two data units to a transmission buffer area corresponding to a target node; determining the storage location parameter of each data unit in the transmission buffer area; obtaining the target write location parameter of each data unit on the target node; and generating metadata corresponding to each data unit based on the storage location parameter and the target write location parameter of each data unit.

[0083] This refers to a data storage area used to temporarily store data units to be transmitted. For example, it can be the local memory area, video memory area, or other storage space used to cache data before data transmission in a computer device. It is used to store data units that have been compressed and quantized so that the network card can read and execute data transmission operations later.

[0084] Storage location parameters are parameters used to characterize the storage location information of a data unit in the transmit buffer area. For example, they can be the starting address, offset, or memory pointer of the data unit in the transmit buffer area, which are used to instruct the network card to read the data of the corresponding data unit from the transmit buffer area.

[0085] The target write location parameter refers to a parameter used to characterize the target storage location of the data unit in the target node. For example, it can be the target address, offset, or target buffer identifier in the memory of the target node, which is used to instruct the network card to write the data unit to the corresponding storage location of the target node during data transmission.

[0086] Specifically, the computer device can write at least two data units into the transmit buffer area corresponding to the target node for storage. Then, for each data unit, the computer device can determine its storage location parameter in the transmit buffer area, where the storage location parameter characterizes the starting address or offset position of the corresponding data unit in the transmit buffer area. Further, the computer device can obtain the target write location parameter of each data unit in the target node, where the target write location parameter characterizes the target write address or offset position of the corresponding data unit in the target node's memory. After obtaining the storage location parameter and the target write location parameter, the computer device can generate metadata corresponding to each data unit based on the storage location parameter and the target write location parameter. The metadata describes the transmission information required during data transmission, allowing the network interface card (NIC) to read the data of the corresponding data unit from the transmit buffer area according to the metadata and write it to the target write location of the target node.

[0087] In one embodiment, the computer device can obtain target buffer area information pre-allocated by the target node. This target buffer area information includes the starting address of the target buffer area, the writable space size, and the write offset information corresponding to each data unit. Subsequently, the computer device can determine the write offset of each data unit in the target buffer area based on its corresponding data identifier, data length, or preset write rule. Further, the computer device can determine the target write position parameter of each data unit in the target node based on the starting address and write offset of the target buffer area. For example, if the target node reserves a continuous target buffer area for a sequence of data units, the computer device can calculate the write offset of each data unit sequentially based on its data identifier and use "target buffer area starting address + write offset" as the target write position parameter for that data unit.

[0088] In the above embodiments, the computer device sends at least two data units to the transmit buffer area corresponding to the target node, determines the storage location parameters of each data unit in the transmit buffer area, obtains the target write location parameters of each data unit in the target node, and generates metadata corresponding to each data unit based on the storage location parameters and target write location parameters of each data unit. This provides complete transmission description information for the network card to perform data transmission, enabling the network card to accurately determine the read and write locations of the data units based on the metadata, and then directly read data from the transmit buffer area and write it to the target node, reducing the address resolution overhead during data transmission and improving the accuracy and efficiency of data transmission.

[0089] In one embodiment, the compression and quantization processing of at least two raw data units is performed by a first thread group; the sorting of at least two data units and the writing of metadata to the transmission queue are performed by a second thread group, with the first and second thread groups executing in parallel; the process of the computer device sending a processing notification to the network card after completing the writing of metadata for all data units specifically includes the following steps: after the second thread group has completed the writing of metadata for all data units and received the quantization completion notification sent by the first thread group, it sends a processing notification to the network card; wherein, the quantization completion notification is sent by the first thread group after completing the compression and quantization processing of all raw data units.

[0090] Here, "thread" refers to one or more threads used to perform raw data unit compression and quantization processing. For example, it can be a set of threads allocated in a graphics processing unit (GPU) for performing token quantization, data compression, or precision conversion.

[0091] The second thread group refers to one or more threads used to perform data unit sorting and metadata writing operations. For example, it can be a set of threads allocated in a graphics processing unit (GPU) to determine the data unit transmission order, generate queue elements, and write the queue elements to the send queue.

[0092] The parallel execution of the first and second thread groups means that while the first thread group is performing compression and quantization processing, the second thread group can perform processing such as data unit sorting, data write position calculation, or metadata writing. The two thread groups do not block each other except for necessary dependency waiting, thereby improving the utilization of graphics processing unit (GPU) computing resources.

[0093] A quantization completion notification is a status indication message sent by the first thread group to the second thread group after the first thread group has completed the compression and quantization processing of all raw data units in the current batch. It is used to indicate that the second thread group can trigger network card processing after the metadata is written. For example, a quantization completion notification can be implemented by setting a memory flag in global memory.

[0094] Specifically, the computer device can divide the computing resources used to perform data transmission tasks into a first thread group and a second thread group. The first thread group is used to obtain the raw data units of the current batch and perform compression and quantization processing on each raw data unit to obtain the corresponding data unit. The second thread group is used to obtain the transmission priority of each data unit, sort each data unit, and generate metadata based on the sorting result and write it into the sending queue.

[0095] In one embodiment, the first thread group and the second thread group can execute in parallel. During the compression and quantization process performed by the first thread group, the second thread group can synchronously perform operations such as sorting, data write position calculation, and metadata writing. After completing the compression and quantization process of all raw data units in the current batch, the first thread group can send a quantization completion notification by setting a memory flag in global memory. After completing the metadata writing of all data units into the transmission queue, the second thread group checks the memory flag. When a quantization completion notification is detected, the second thread group sends a processing notification to the network card to trigger the network card to process the queue elements in the transmission queue, thereby preventing the network card from reading data that has not yet been quantized.

[0096] Combination Figure 5The proposed solution is illustrated below. The Quantization & NVLink thread group on the left represents the GPU computing resource thread group used for quantizing token data and transmitting it via NVLink. The RDMA thread group below represents the thread group used for sorting token data, collaboratively writing to WQE (queue element), and finally notifying the network card. The timeline shows the execution order of each stage, including token quantization, notification of quantization completion, NVLink transmission, token sorting, collaborative writing to WQE, waiting for quantization completion, and sending WQE notification to the network card, as shown in the figure. This solution incorporates the DeepEP LL Dispatch sending phase (LL... To adapt the LL send process to a sequence-sensitive, low-queue scenario, the GPU is divided into two parallel thread groups: the quantization & NVLink thread group is responsible for token compression quantization and NVLink transmission, and the RDMA thread group is responsible for token sorting and WQE writing. Before notifying the network card, the RDMA thread group needs to wait unidirectionally for the quantization & NVLink thread group to complete token quantization to avoid the network card processing unquantized data. The quantization & NVLink thread group does not need to wait for the RDMA thread group; this waiting is achieved by setting a memory flag in GMEM. Apart from the necessary unidirectional waiting, the two thread groups execute completely in parallel without interfering with each other. Combined with the contention-free WQE issuance mechanism, the RDMA thread group can first sort the tokens in high-speed storage SMEM, reducing access to GMEM and achieving fast WQE issuance. After the WQE writing is completed, the network card is notified in a unified manner, ensuring order and reducing operation latency. In summary, this solution effectively shortens the data issuance time in the LL send stage and improves the transmission efficiency in sequence-sensitive scenarios by decoupling WQE writing from network card notification, parallel GPU execution, and SMEM sorting optimization.

[0097] In the above embodiments, the computer device distributes the compression and quantization processing, data unit sorting, and metadata writing operations to a first thread group and a second thread group that are executed in parallel. After the first thread group completes all compression and quantization processing, it sends a quantization completion notification to the second thread group. This allows the second thread group to send a processing notification to the network card only after completing metadata writing and confirming that quantization is complete. This ensures that the network card does not read data that has not been quantized, thereby achieving parallel execution of quantization processing and sending queue writing processes, reducing serial waiting time, and improving the utilization of computing resources and data transmission efficiency.

[0098] In one embodiment, the above data transmission method further includes the following steps: determining the target thread for data transmission; determining the data unit to be processed corresponding to the target thread in the data unit sequence; the process of the computer device writing the metadata corresponding to the data unit in the data unit sequence into the sending queue according to the data writing position specifically includes the following steps: the target thread writes the metadata corresponding to the data unit to be processed into the corresponding queue slot in the sending queue according to the data writing position corresponding to the data unit to be processed.

[0099] The target thread refers to the execution unit assigned to perform data transmission related operations. For example, it can be a thread or thread instance in the central processing unit or graphics processor. It is used to process some data units and perform corresponding metadata writing operations according to preset rules, such as thread 0, thread 1, etc. Each target thread is responsible for processing some data units and performing corresponding metadata writing operations.

[0100] The data unit to be processed corresponding to the target thread refers to the set of data units that are assigned to a target thread for processing according to a preset allocation rule. The data units in this set are processed independently by the target thread and the corresponding metadata writing operation is completed. For example, when there are multiple data units and the number of target threads is 2, the data units can be divided according to the thread identifier and the number of threads, so that thread 0 processes data units with odd numbers such as data unit 1, data unit 3, and data unit 5, and thread 1 processes data units with even numbers such as data unit 2, data unit 4, and data unit 6, so that different target threads process different sets of data units.

[0101] A queue slot is a basic storage unit in a sending queue used to store a single queue element. Each queue slot corresponds to a unique position index and is used to store the metadata corresponding to a data unit, thus forming a sequence of queue elements in the sending queue. For example, the nth storage position in the sending queue (such as the kth, k+1th, or k+2th position) is used to store the metadata corresponding to a data unit. Each queue slot corresponds to a unique index value. Different target threads can write the metadata to different queue slots according to the data write position, thereby avoiding write conflicts between threads.

[0102] Specifically, the computer device can determine at least one target thread for data transmission. For any target thread, the data unit to be processed corresponding to the target thread is determined from the data unit sequence according to a preset allocation rule. After determining all the data units to be processed corresponding to the target thread, the target thread can obtain the data write position corresponding to each data unit to be processed, and write the metadata of the corresponding data unit into the corresponding queue slot in the sending queue according to the data write position to complete the queue element write operation. By performing the above processing on each target thread separately, multiple target threads can independently complete data writing without the need for inter-thread lock control or atomic operations, realizing the parallel construction of the sending queue, avoiding the additional overhead caused by inter-thread competition, and improving the queue element writing efficiency and overall data transmission performance.

[0103] In the above embodiments, the computer device determines the target thread for data transmission, identifies the data unit to be processed corresponding to the target thread in the data unit sequence, and the target thread writes the metadata corresponding to the data unit to be processed into the corresponding queue slot in the sending queue according to the data writing position corresponding to the data unit to be processed. This enables different target threads to independently complete the metadata writing operation according to the predetermined data writing position, avoiding the competition and synchronization overhead between threads for the writing position of the sending queue, realizing the parallel construction of queue elements, and improving data writing efficiency and overall data transmission performance.

[0104] In one embodiment, the process by which a computer device determines the data unit to be processed corresponding to a target thread in a data unit sequence includes the following steps: obtaining the thread identifier and thread count of the target thread; for each target thread, determining the data unit to be processed corresponding to the target thread based on the thread identifier and thread count; selecting the data unit corresponding to the data unit to be processed in the data unit sequence; wherein the selected data unit is used as the data unit to be processed for the target thread.

[0105] The thread identifier is a unique identifier used to distinguish different target threads. For example, it can be the number of the thread in the current thread group, such as 0, 1, 2, ..., T-1, where T represents the total number of threads. Each thread identifier corresponds to a target thread.

[0106] The number of threads refers to the total number of target threads participating in the current data transmission and processing. For example, it can be 2, 4, 8, etc., representing the number of threads used to process data units in parallel.

[0107] The data identifier to be processed refers to the information used to characterize the position of the data unit that a certain target thread needs to process, determined according to the thread identifier and the number of threads and a preset allocation rule. For example, in the data unit sequence, when the number of threads is 2, the target thread with thread identifier 0 can process data units with data identifiers 1, 3, 5, ..., and the target thread with thread identifier 1 can process data units with data identifiers 2, 4, 6, ..., so that different target threads process data units at different positions.

[0108] Specifically, the computer device can obtain the thread identifier and thread count of the target threads participating in the current data transmission task. The thread identifier is used to distinguish different target threads, and the thread count represents the total number of data transmission threads participating in parallel processing. For any target thread, the computer device can determine the data identifier to be processed corresponding to the target thread based on the thread identifier and thread count of the target thread according to a preset allocation rule. By performing the above processing on each target thread, the set of data identifiers to be processed corresponding to each target thread can be determined. For any target thread, the corresponding data unit is selected from the data unit sequence based on the set of data identifiers to be processed corresponding to the target thread, and the selected data unit is used as the data unit to be processed corresponding to the target thread. By performing the above processing on each target thread, the data unit to be processed corresponding to each target thread can be determined.

[0109] In one embodiment, for target thread i, the identifier of the data to be processed corresponding to target thread i can be determined in the following way:

[0110]

[0111] Where m is the identifier of the data to be processed that is assigned to target thread i in the data unit sequence; T is the total number of target threads participating in the current data transmission task; h = 0, 1, 2, ..., is used to iteratively calculate all the identifiers of the data to be processed assigned to target thread i, thereby covering the set of data units that the thread should process.

[0112] In the above embodiments, the computer device obtains the thread identifier and thread count of the target thread. For each target thread, based on the thread identifier and thread count, it determines the data identifier to be processed corresponding to the target thread. In the data unit sequence, it selects the data unit corresponding to the data identifier to be processed as the data unit to be processed of the target thread. This enables each target thread to independently determine its set of data units to be processed according to a preset allocation rule, avoiding repeated processing of the same data unit by multiple threads, achieving uniform distribution of data units among different threads, and completing the data allocation process without the need for inter-thread synchronization or atomic operations. This reduces the overhead of inter-thread competition and improves the efficiency of parallel processing.

[0113] In one embodiment, the number of threads is at least two, and at least two target threads execute in parallel when writing the metadata corresponding to the data unit to be processed to the sending queue; the above data transmission method further includes the following steps: after each target thread completes the writing of the metadata of all data units to be processed, it performs a memory barrier operation to make the written metadata visible to the network card; the process of the computer device sending a processing notification to the network card after completing the writing of the metadata of all data units includes the following steps: after all target threads complete the writing of the metadata of all data units and the memory barrier operation, one of the at least two target threads sends a processing notification to the network card.

[0114] The parallel execution of at least two target threads writing the metadata corresponding to the data unit to be processed into the sending queue means that within the same time period, multiple target threads simultaneously write the metadata of the corresponding data unit into different queue slots in the sending queue according to their respective data writing positions. The target threads do not need to be synchronized through lock control or atomic operations, thereby achieving parallel construction of the sending queue. For example, when the number of threads is 2, thread 0 can write the metadata corresponding to data unit 1, data unit 3, and data unit 5 into the k-th, k+2-th, and k+4-th queue slots in the sending queue, respectively. Thread 1 can write the metadata corresponding to data unit 2, data unit 4, and data unit 6 into the k+1-th, k+3-th, and k+5-th queue slots in the sending queue, respectively. Here, k represents the starting writing position of the sending queue. Thread 0 and thread 1 execute in parallel during the writing process of their respective queue slots, and there is no need for them to wait or perform synchronization operations.

[0115] A memory barrier operation is a memory access order control operation performed by the target thread after the metadata is written. It is used to ensure that all metadata written to the send queue has been committed to memory and is visible to the network card before the memory barrier operation is performed. This prevents the network card from reading data that has not been written when processing queue elements, thereby ensuring the consistency and correctness of data transmission. For example, after multiple target threads write the metadata of their respective data units to the send queue, each target thread performs a memory barrier operation after completing the write operation, so that all queue elements in the send queue have been flushed to memory and are visible to the network card.

[0116] Specifically, the computer device can use multiple target threads to write the metadata of their respective data units to be processed into the corresponding queue slots in the transmission queue. After each target thread completes the writing of the metadata of all its corresponding data units to be processed, a memory barrier operation is performed to ensure that the metadata written by the target thread has been committed to memory and is visible to the network card. Furthermore, the computer device can synchronously confirm the writing completion status of all target threads. When it is detected that all target threads have completed the writing of the metadata of all data units and have all performed the memory barrier operation, one of the at least two target threads sends a processing notification to the network card to trigger the network card to process the queue elements in the transmission queue.

[0117] In one embodiment, notification processing can be achieved by writing notification information to the doorbell register of the network card or updating the doorbell record, thereby enabling the network card to start reading queue elements in the transmission queue in sequence and performing the corresponding data transmission operations.

[0118] In the above embodiments, the computer device enables multiple target threads to complete the writing of metadata for their respective data units and perform memory barrier operations. After confirming that all target threads have completed the writing and that the metadata is visible to the network card, one of the threads sends a processing notification to the network card. This avoids the serial overhead caused by triggering the network card processing multiple times after each data unit is written, reduces the number of notifications and lowers system latency, while ensuring that the data read by the network card is complete and ordered, thus improving the efficiency and reliability of data transmission.

[0119] In one embodiment, the process by which a computer device determines the data write position based on a data identifier specifically includes the following steps: obtaining an initial value of a counter used to characterize the starting write position of the sending queue; and determining the data write position corresponding to each data unit based on the data identifier and the initial value of the counter.

[0120] The starting write position of the sending queue refers to the starting storage position of the next batch of queue elements to be written in the current sending queue. That is, it is the position parameter used to identify the first queue slot in the sending queue where the current write operation begins. For example, when the first 50 queue slots in the sending queue have been occupied, the 51st queue slot can be used as the starting write position of the current sending queue. During the writing of this batch of data, the metadata corresponding to each data unit can be written to the subsequent queue slots in sequence from this starting write position.

[0121] The initial value of the counter refers to the current value of the counter used to record the starting write position of the send queue. Specifically, it can be the number of queue slots that have been occupied in the send queue or the position of the current write pointer. For example, when 50 queue slots in the send queue have been occupied, the initial value of the counter can be 50, and the corresponding starting write position is the 51st queue slot. Based on this, each data unit can calculate the corresponding data write position based on its corresponding data identifier and the initial value of the counter.

[0122] Specifically, the computer device can obtain an initial value of a counter used to characterize the starting write position of the sending queue. The initial value of the counter is used to indicate the starting slot where the next batch of data in the current sending queue begins to be written. The computer device obtains the data identifier corresponding to each data unit. Based on the initial value of the counter and the data identifier, the computer device determines the data writing position corresponding to each data unit through a preset mapping rule. The data writing position can be obtained by combining the initial value of the counter and the data identifier, thereby establishing a correspondence between the data unit and the storage slot in the sending queue.

[0123] In one optional implementation, when the initial value of the counter is k, the data writing positions corresponding to each data unit with data identifiers 1, 2, 3, ... can be k, k+1, k+2, ... respectively, so that each data unit can be written into the consecutive queue slots in the transmission queue in accordance with its transmission order.

[0124] It should be noted that in a multi-threaded scenario, different target threads can independently calculate the data write position based on their respective data identifiers and write the metadata to different queue slots in the send queue, thereby avoiding write conflicts between threads and improving parallel write efficiency.

[0125] In the above embodiments, the computer device obtains the initial value of the counter used to characterize the starting write position of the sending queue, and determines the data write position corresponding to each data unit based on the data identifier and the initial value of the counter. Thus, the allocation of queue slots can be completed without dynamically obtaining the write position through locking mechanisms or atomic operations, so that different data units can be directly mapped to the corresponding storage slots in the sending queue, realizing deterministic calculation of the write position, avoiding inter-thread competition and improving the efficiency of multi-threaded parallel writing, while ensuring that data units are written to the sending queue in a preset transmission order.

[0126] In one embodiment, the above data transmission method further includes the following steps: after sending a processing notification to the network card, obtaining the maximum position parameter of the written queue element corresponding to the sending queue; updating the initial value of the counter based on the maximum position parameter to obtain the updated initial value of the counter.

[0127] Among them, the queue element that has been written to the queue refers to the queue element that has been written to the corresponding storage slot in the sending queue by the computer device during the current batch of data transmission. For example, after writing the metadata corresponding to 10 data units into the sending queue in the current batch, the queue element composed of these 10 metadata units is the queue element that has been written to the queue.

[0128] The maximum position parameter refers to the maximum position index value of the queue element that has been written to the sending queue. That is, the index of the last queue slot among all queue elements written in the current batch. For example, when the queue elements written in this batch occupy slots k to k+9 in the sending queue, the maximum position parameter can be k+9.

[0129] Updating the initial value of the counter means updating the current value of the counter to the starting position for writing the next batch of data based on the maximum position parameter. For example, updating the initial value of the counter to the maximum position parameter plus 1 will allow the metadata of the next batch of data units to continue to be written to the sending queue from the updated starting position.

[0130] Specifically, after sending a processing notification to the network card, the computer device can determine the range of queue elements that have been written into the transmission queue in the current batch. Subsequently, the computer device obtains the maximum position parameter corresponding to the queue elements written into the queue, where the maximum position parameter is used to characterize the maximum slot index of the queue elements written in the current batch in the transmission queue. Further, the computer device can update the initial value of the counter based on the maximum position parameter, for example, by updating the initial value of the counter to the maximum position parameter, thereby obtaining the updated initial value of the counter.

[0131] In one embodiment, when the queue elements written in the current batch occupy slots k to k+n in the sending queue, the maximum position parameter can be k+n, and the initial value of the updated counter can be k+n+1, used to indicate the starting position of writing the metadata of the next batch of data units into the sending queue. In this way, the computer device can continuously update the start position of writing in the sending queue after completing the writing and processing of the current batch of data, thereby ensuring that subsequent batches of data can be written sequentially into the sending queue.

[0132] In the above embodiments, after sending a processing notification to the network card, the computer device obtains the maximum position parameter of the written queue element corresponding to the sending queue, updates the initial value of the counter based on the maximum position parameter, and obtains the updated initial value of the counter. This allows the writing start position of the sending queue to continuously advance with the data batches that have been written, avoiding subsequent data writing from overwriting existing queue elements, ensuring the continuity and order of different batches of data in the sending queue, reducing frequent update operations on the global counter, and improving data writing efficiency and overall system performance.

[0133] Combination Figure 6 The following describes the solution proposed in this application. In the diagram, "(a) Current technology writes WQE" represents the process of writing queue elements in the existing communication library; "(b) This application writes WQE" represents the optimized process of writing queue elements in this application; "(c) Current technology notifies the network card" represents the process of sending a processing notification to the network card after writing WQE in the existing communication library; and "(d) This application notifies the network card" represents the process of sending a processing notification to the network card after writing WQE in this application. GMEM is global memory, which contains the WQE for each QP. The buffer and three counters: resv, ready, and prod, along with the SMEM / register, represent the high-speed storage available in this solution, used to store WQE information and a distribution order table, as shown in the figure. In existing technologies, when writing to WQE (corresponding to (a)), atomic operations are required to access counters located in GMEM. First, an atomic increment operation is performed on the resv counter to obtain the WQE write index; after writing, CAS operations are used repeatedly to update the ready counter, forming a spinlock and creating waiting between threads; finally, when notifying the network card (corresponding to (c)), further... To update the prod counter, these atomic operations and cyclic updates increase the time spent writing WQE. In the solution of this application (corresponding to (b)), each thread can directly write WQE to a specified index without competing with other threads. To ensure the correctness of the write, at the beginning of the batch distribution cycle, the thread can first save the counter information in SMEM or a register and calculate the WQE index independently based on a preset algorithm to ensure that the WQE written by each thread does not conflict. Subsequently (corresponding to (d)), after all WQE writes are completed, the network card is notified to process them in a unified manner, thereby significantly reducing thread competition and write latency.

[0134] In one embodiment, the target node is an expert node in a distributed system; the number of expert nodes is at least two, and each expert node corresponds to a sending queue; the sending queues corresponding to the at least two expert nodes are used to transmit data to the at least two expert nodes in parallel.

[0135] This refers to a system composed of multiple interconnected computing nodes that can logically collaborate to complete a unified computing task. Each computing node can independently perform computation, storage, and data processing operations, and interact with data and collaborate on tasks through the network, thereby achieving high-performance computing capabilities and scalability of the overall system. For example, in the training or inference scenarios of large artificial intelligence models, a distributed system can consist of multiple server nodes equipped with graphics processors. Each node is responsible for calculating model parameters, processing data, or transmitting intermediate results, and communicates and synchronizes through a high-speed network.

[0136] An expert node is a computing node in a distributed system that is used to perform specific computing tasks or process specific subsets of data. Each expert node can correspond to an independent processing unit or computing resource for specialized processing of the data allocated to it. For example, in a large-scale artificial intelligence model that adopts an expert parallel structure, each expert node can correspond to a different expert model or sub-model. Each expert node is responsible for processing the data unit (such as a token) allocated to that expert and outputting the corresponding computing results, thereby realizing parallel computing by multiple experts to improve overall processing efficiency.

[0137] Each expert node has its own sending queue, meaning that at the data sending end, an independent sending queue is established for each target expert node to store the queue element corresponding to the data unit sent to that expert node. This ensures that data transmission between different expert nodes is independent of each other. Figure 7 As shown, in the presence of multiple expert nodes, the sending end can configure sending queue A, sending queue B, and sending queue C for expert node 0, expert node 1, and expert node 2, respectively. The metadata corresponding to the data unit sent to expert node 0 is written to sending queue A, the metadata corresponding to the data unit sent to expert node 1 is written to sending queue B, and the metadata corresponding to the data unit sent to expert node 2 is written to sending queue C, so that each sending queue corresponds to different expert nodes and performs data transmission in parallel.

[0138] Specifically, the computer device can identify at least two target nodes as expert nodes in a distributed system and establish a corresponding sending queue for each expert node. Subsequently, the computer device can divide the data units to be transmitted into the sending queues of the corresponding expert nodes according to the target nodes of each data unit. Furthermore, the computer device can execute the write operation of queue elements in parallel through multiple target threads, writing the metadata corresponding to each data unit into the corresponding queue slots in the sending queues of different expert nodes, thereby realizing the parallel construction of multiple sending queues. After completing the writing of queue elements in each sending queue, the computer device can send processing notifications to the network card transmission channels corresponding to each sending queue, enabling the network card to execute data transmission operations in parallel based on different sending queues, and sending the corresponding data units to each expert node respectively.

[0139] In the above embodiments, the computer device establishes corresponding transmission queues for at least two expert nodes, and writes the metadata corresponding to the data unit into the corresponding queue slot in each transmission queue in parallel through multiple target threads. After the writing is completed, the network card is triggered to perform data transmission operations in parallel based on multiple transmission queues, thereby realizing dual parallel processing of queue element writing and data transmission, avoiding the performance bottleneck caused by single queue serial processing, and improving the data transmission throughput and overall system performance in multi-node scenarios.

[0140] In one embodiment, such as Figure 8 As shown, a data transmission method is also provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0141] S802, acquire at least two raw data units to be transmitted for the target node; perform compression and quantization processing on the at least two raw data units respectively to obtain at least two data units to be transmitted.

[0142] S804, obtain the transmission priority corresponding to each data unit; sort at least two data units according to the transmission priority to obtain the data unit sequence.

[0143] S806: Send at least two data units to the send buffer area corresponding to the target node; determine the storage location parameter of each data unit in the send buffer area; obtain the target write location parameter of each data unit in the target node; and generate metadata corresponding to each data unit based on the storage location parameter and the target write location parameter of each data unit.

[0144] S808: Determine the data identifier of each data unit according to the sorting position of each data unit in the data unit sequence; obtain the initial value of the counter used to characterize the starting write position of the sending queue; determine the data write position corresponding to each data unit based on the data identifier and the initial value of the counter.

[0145] S810, determine the target thread for data transmission; obtain the thread identifier and thread count of the target thread; for each target thread, determine the data identifier to be processed corresponding to the target thread based on the thread identifier and thread count; select the data unit corresponding to the data identifier to be processed from the data unit sequence; wherein the selected data unit is used as the data unit to be processed of the target thread.

[0146] S812, through the target thread, writes the metadata corresponding to the data unit to be processed into the corresponding queue slot in the sending queue according to the data writing position corresponding to the data unit to be processed.

[0147] S814, after completing the writing of metadata for all data units, sends a processing notification to the network card by one of the at least two target threads, so that the network card, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

[0148] This application also provides an application scenario, specifically the DeepEP LL Dispatch transmission (LLsend) scenario, which applies the aforementioned data transmission method, referencing... Figure 9The schematic diagram shown and Figure 10 The timing diagram shown illustrates that this data transmission method includes the following steps:

[0149] 1. Obtain data units and data identifiers.

[0150] The computer device acquires the sequence of data units to be transmitted and the data identifier corresponding to each data unit to characterize the transmission order, including the target write position and transmission priority.

[0151] 2. Determine the data writing location based on the data identifier.

[0152] The computer device determines the queue slot corresponding to each data unit based on the data identifier of each data unit and the starting write position of the transmission queue.

[0153] 3. Write metadata to the send queue.

[0154] The computer device writes the metadata of each data unit (including data source address, length, target write position, operation type, etc.) as a queue element to the corresponding queue slot in the send queue. If there are multiple target threads, queue elements in different slots can be written in parallel.

[0155] 4. Memory barrier operation.

[0156] After each target thread has completed writing the metadata of all pending data units, it performs a memory barrier operation to ensure that all queue elements are visible to the network interface card.

[0157] 5. Send a processing notification to the network card.

[0158] After all queue element writes and memory barrier operations are completed, one or more target threads send a processing notification to the network interface card (NIC) to trigger the NIC to perform unified processing on the queue elements in the transmit queue.

[0159] 6. Network interface card (NIC) processes queue elements.

[0160] After receiving the processing notification, the network card performs the corresponding operation according to the operation type of each queue element: for the Write operation, the data unit is completely written to the target node's memory; for the FetchAdd operation, an atomic addition is performed on the counter or status of the target node at the specified memory address.

[0161] 7. Data transmission complete.

[0162] The network interface card (NIC) completes the writing and atomic operations to the target node according to the data identifiers in the queue elements, realizing order-sensitive distributed data transmission.

[0163] Taking the implemented TRMT-DeepEP product as an example, compared to the original DeepEP architecture, this solution can complete the send phase request delivery in a specific order with fewer QPs (queue pairs). Despite the reduced number of QPs, the total time spent in the send phase not only does not increase but actually decreases. Specifically, this solution can complete the WQE delivery with fewer QPs while meeting the order-sensitive requirements; it maintains the send phase time without increasing the order-sensitive requirements. Experimental results show that this solution not only maintains the same time but also takes less time than the original LL send. The experimental data and charts below mainly illustrate this quantitative effect. Figure 11 , Figure 11 This demonstrates the performance improvement of the proposed solution (dispatch_1) in the send stage under different batch sizes. The experimental environment consisted of 16 GPUs, with 16 expert nodes on each GPU. Dispatch_1 used only 2 QPs per destination GPU, while the original solution used 16 QPs. The results show that despite the significant reduction in the number of QPs used, the proposed solution's execution time was lower than the original LL send at all batch sizes. Furthermore, the execution time decreased further as the batch size increased, showing a more significant effect. (Reference) Figure 12 , Figure 12 This demonstrates the contribution of each technical aspect to performance, and the experimental conditions and... Figure 11 Similarly, the batch size for each GPU is 128. In the diagram, "original version" means that DeepEP's original LL send uses 16 QPs. "Sorted and then sent" means that the QPs are reduced directly and WQEs are sent in order. This method is time-consuming in scenarios where the order of 1 to 4 QPs is sensitive. "Sorted + batch no-contention sending" means that when using more than 2 QPs, the time consumption is reduced to below that of the original version. However, when the QPs are further reduced, the time consumption still increases. "This solution" further reduces the time consumption and is almost unaffected by changes in the number of QPs.

[0164] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0165] Based on the same inventive concept, this application also provides a data transmission apparatus for implementing the data transmission method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, specific limitations in one or more data transmission apparatus embodiments provided below can be found in the limitations of the data transmission method described above, and will not be repeated here.

[0166] In one embodiment, such as Figure 13 As shown, a data transmission device is provided, including: a data identifier acquisition module 1302, a write location determination module 1304, a metadata writing module 1306, and a data execution module 1308, wherein:

[0167] The data identifier acquisition module 1302 is used to acquire the data identifier of each data unit in the data unit sequence, which is used to characterize the transmission order;

[0168] The write location determination module 1304 is used to determine the data write location based on the data identifier;

[0169] The metadata writing module 1306 is used to write the metadata corresponding to the data unit in the data unit sequence into the sending queue according to the data writing position;

[0170] The data execution module 1308 is used to send a processing notification to the network card after writing the metadata of all data units, so that the network card can execute the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier after receiving the processing notification.

[0171] In the above embodiments, by obtaining the data identifiers of each data unit in the data unit sequence to characterize the transmission order, determining the data writing position based on the data identifiers, and writing the metadata corresponding to the data units in the data unit sequence into the transmission queue according to the data writing positions, and after completing the writing of the metadata of all data units, a processing notification is sent to the network card, so that after receiving the processing notification, the network card executes the data units corresponding to the metadata in the transmission queue to the target node based on the data identifiers. Thus, an ordered arrangement relationship of data units is pre-established in the transmission queue according to the data identifiers, enabling the network card to process each data unit according to the transmission priority order. At the same time, the network card is notified to process after all metadata is written, reducing the number of notifications and the latency overhead caused by serial triggering, avoiding multiple waiting and processing blockages, thereby improving the data transmission efficiency in sequence-sensitive scenarios and enhancing the parallel processing capability and overall performance of the system.

[0172] In one embodiment, such as Figure 14As shown, the device also includes a sorting module 1310, which is used to: acquire at least two data units to be transmitted for the target node; acquire the transmission priority corresponding to each data unit; sort the at least two data units according to the transmission priority to obtain a data unit sequence; and the data identifier acquisition module 1302 is also used to: determine the data identifier of each data unit according to the sorting position of each data unit in the data unit sequence.

[0173] In one embodiment, the sorting module 1310 is further configured to: obtain at least two raw data units to be transmitted for the target node; and perform compression and quantization processing on the at least two raw data units respectively to obtain at least two data units to be transmitted.

[0174] In one embodiment, such as Figure 14 As shown, the device also includes a metadata generation module 1312, used for: sending at least two data units to a send buffer area corresponding to a target node; determining the storage location parameters of each data unit in the send buffer area; obtaining the target write location parameters of each data unit in the target node; and generating metadata corresponding to each data unit based on the storage location parameters and target write location parameters of each data unit.

[0175] In one embodiment, the compression and quantization processing of at least two raw data units is performed by a first thread group; the sorting of at least two data units and the writing of metadata to the transmission queue are performed by a second thread group, and the first thread group and the second thread group are executed in parallel; the data execution module 1308 is further configured to: after the second thread group has completed the writing of metadata for all data units and received the quantization completion notification sent by the first thread group, send a processing notification to the network card; wherein the quantization completion notification is sent by the first thread group after completing the compression and quantization processing of all raw data units.

[0176] In one embodiment, such as Figure 14 As shown, the device also includes a thread allocation module 1314, used to: determine the target thread for data transmission; determine the data unit to be processed corresponding to the target thread in the data unit sequence; and a metadata writing module 1306, used to: write the metadata corresponding to the data unit to be processed into the corresponding queue slot in the sending queue according to the data writing position corresponding to the data unit to be processed by the target thread.

[0177] In one embodiment, the apparatus further includes a thread allocation module 1314, which is further configured to: obtain the thread identifier and thread count of the target thread; for each target thread, determine the data identifier to be processed corresponding to the target thread based on the thread identifier and thread count; select the data unit corresponding to the data identifier to be processed from the data unit sequence; wherein the selected data unit is used as the data unit to be processed of the target thread.

[0178] In one embodiment, the number of threads is at least two, and the at least two target threads execute in parallel when writing the metadata corresponding to the data unit to be processed into the sending queue; the metadata writing module 1306 is further configured to: after each target thread completes the writing of the metadata of all data units to be processed, perform a memory barrier operation to make the written metadata visible to the network card; the data execution module 1308 is further configured to: after all target threads complete the writing of the metadata of all data units and the memory barrier operation, one of the at least two target threads sends a processing notification to the network card.

[0179] In one embodiment, the write position determination module 1304 is further configured to: obtain an initial value of a counter used to characterize the start write position of the sending queue; and determine the data write position corresponding to each data unit based on the data identifier and the initial value of the counter.

[0180] In one embodiment, the metadata writing module 1306 is further configured to: after sending a processing notification to the network card, obtain the maximum position parameter of the written queue element corresponding to the sending queue; update the initial value of the counter based on the maximum position parameter to obtain the updated initial value of the counter.

[0181] In one embodiment, the target node is an expert node in a distributed system; the number of expert nodes is at least two, and each expert node corresponds to a sending queue; the sending queues corresponding to the at least two expert nodes are used to transmit data to the at least two expert nodes in parallel.

[0182] Each module in the aforementioned data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0183] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data to be sent. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data transmission method.

[0184] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0186] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0187] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0189] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data transmission method, characterized in that, The method includes: Obtain the data identifier used to characterize the transmission order of each data unit in the data unit sequence; The data writing location is determined based on the data identifier; According to the data writing position, the metadata corresponding to the data unit in the data unit sequence is written into the sending queue; After writing the metadata of all the data units, a processing notification is sent to the network interface card (NIC) so that the NIC, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

2. The method according to claim 1, characterized in that, The method further includes: Obtain at least two data units to be transmitted for the target node; Obtain the transmission priority corresponding to each data unit; At least two of the data units are sorted according to the transmission priority to obtain a data unit sequence; The data identifier used to characterize the transmission order of each data unit in the acquired data unit sequence includes: The data identifier of each data unit is determined according to the sorting position of each data unit in the data unit sequence.

3. The method according to claim 2, characterized in that, The acquisition of at least two data units to be transmitted for the target node includes: Obtain at least two raw data units to be transmitted for the target node; At least two of the original data units are compressed and quantized respectively to obtain at least two data units to be transmitted.

4. The method according to claim 3, characterized in that, The method further includes: Send at least two of the data units to the transmission buffer area corresponding to the target node; Determine the storage location parameters of each data unit in the transmission buffer area; Obtain the target write position parameters for each data unit at the target node; Based on the storage location parameters of each data unit and the target write location parameters, metadata corresponding to each data unit is generated.

5. The method according to claim 3, characterized in that, Compression and quantization processing of at least two of the original data units is performed through the first thread group; The sorting of at least two of the data units and the writing of the metadata into the send queue are performed by a second thread group, and the first thread group and the second thread group are executed in parallel. After writing the metadata of all the data units, the step of sending a processing notification to the network card includes: After the second thread group has completed writing the metadata of all the data units and received the quantization completion notification sent by the first thread group, it sends a processing notification to the network card. The quantization completion notification is sent by the first thread group after it has completed the compression and quantization processing of all the original data units.

6. The method according to claim 1, characterized in that, The method further includes: Determine the target thread for data transfer; Determine the data unit to be processed corresponding to the target thread in the data unit sequence; The step of writing the metadata corresponding to the data unit in the data unit sequence to the sending queue according to the data writing position includes: The target thread writes the metadata corresponding to the data unit to be processed into the corresponding queue slot in the sending queue according to the data writing position corresponding to the data unit to be processed.

7. The method according to claim 6, characterized in that, The step of determining the data unit to be processed corresponding to the target thread in the data unit sequence includes: Obtain the thread identifier and thread count of the target thread; For each target thread, a data identifier to be processed corresponding to the target thread is determined based on the thread identifier and the number of threads; In the data unit sequence, the data unit corresponding to the data identifier to be processed is selected; wherein, the selected data unit is used as the data unit to be processed by the target thread.

8. The method according to claim 7, characterized in that, The number of threads is at least two, and at least two target threads execute in parallel when writing the metadata corresponding to the data unit to be processed into the sending queue; The method further includes: After each target thread has completed writing the metadata of all the data units to be processed, it performs a memory barrier operation to make the written metadata visible to the network card. After writing the metadata of all the data units, the step of sending a processing notification to the network card includes: After all the target threads have completed the writing of metadata and memory barrier operations for all the data units, one of the at least two target threads sends a processing notification to the network card.

9. The method according to claim 1, characterized in that, Determining the data write location based on the data identifier includes: Obtain the initial value of the counter used to characterize the starting write position of the send queue; The data write position corresponding to each data unit is determined based on the data identifier and the initial value of the counter.

10. The method according to claim 9, characterized in that, The method further includes: After sending a processing notification to the network card, obtain the maximum position parameter of the written queue element corresponding to the sending queue; The initial value of the counter is updated based on the maximum position parameter to obtain the updated initial value of the counter.

11. The method according to any one of claims 1 to 9, characterized in that, The target node is an expert node in the distributed system; The number of expert nodes is at least two, and each expert node corresponds to a sending queue; The sending queues corresponding to at least two of the expert nodes are used to transmit data to the at least two expert nodes in parallel.

12. A data transmission device, characterized in that, The device includes: The data identifier acquisition module is used to acquire the data identifier of each data unit in the data unit sequence, which is used to represent the transmission order; A write location determination module is used to determine the data write location based on the data identifier; The metadata writing module is used to write the metadata corresponding to the data unit in the data unit sequence into the sending queue according to the data writing position; The data execution module is used to send a processing notification to the network interface card (NIC) after writing the metadata of all the data units, so that the NIC, upon receiving the processing notification, executes the data unit corresponding to the metadata in the sending queue to the target node based on the data identifier.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images