Data communication method and processor
By constructing RDMA channels and circular queue buffer areas in a distributed cluster, a block-based pipelined transmission and notification mechanism for data blocks is implemented, which solves the communication bottleneck of the MoE model in a distributed scenario and improves the training and inference efficiency of large language models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2026-01-29
- Publication Date
- 2026-04-17
AI Technical Summary
In distributed training or inference scenarios, hybrid expert models (MoE) bring significant pain points in communication requirements, manifested in complex communication modes, severe network congestion, and significant latency bottlenecks, resulting in high-performance communication bottlenecks that mask computational advantages.
By establishing RDMA channels between processors in a distributed cluster and building a circular queue as a buffer area on each channel, a block-based pipelined transmission and notification mechanism for data blocks is implemented, avoiding intermediate copying and improving communication efficiency.
It significantly improves the data transmission efficiency between processors in a distributed cluster, reduces latency, enhances model training or inference efficiency, and solves the communication needs pain point of MoE.
Smart Images
Figure CN121880255A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and communication technology, and in particular to a data communication method and processor. Background Technology
[0002] Mixture-of-experts (MoE) models have been widely used in large language models to scale up to trillions of parameters while maintaining a fixed computational cost. MoE decouples parameter size from computational cost by activating only a small number of experts in each forward computation, thereby controlling computational overhead while maintaining model size.
[0003] However, in distributed training or inference scenarios, MoE presents significant communication challenges, with the core challenge shifting from computational bottlenecks to high-performance communication bottlenecks. Solving the communication bottleneck problem in large-scale model training or inference in distributed scenarios has become a research hotspot in this field. Summary of the Invention
[0004] Therefore, this application discloses the following technical solution:
[0005] A data communication method applied to a first processor, the method comprising:
[0006] Obtain the word sequence of the target data, and transform the word sequence into multiple data blocks; each data block includes at least one word from the word sequence.
[0007] Based on the data block distribution strategy, each data block is stored in its corresponding first cache area; each first cache area corresponds to a corresponding second processor, and each first cache area is located on the direct memory access channel between the corresponding second processor and the first processor;
[0008] Transmit the target data block currently to be transmitted in the first cache region to the corresponding second processor;
[0009] The corresponding second processor receives a target notification signal for the target data block, the target notification signal indicating that the corresponding second processor has completed receiving the target data block;
[0010] The next data block of the target data block in the first cache region is transmitted to the corresponding second processor.
[0011] Optionally, the first processor and the second processor are different processors in a distributed cluster; the different processors in the distributed cluster are each deployed with different parts of the inference model, so as to perform model training or inference on the received data blocks based on the corresponding parts of the deployed inference model.
[0012] Optionally, the number of the second processors can be multiple;
[0013] Different direct memory access channels are established between different second processors and the first processor; different circular queues are established on the direct memory access channels corresponding to different second processors.
[0014] The first cache area includes the circular queue.
[0015] Optionally, based on the data block distribution strategy, each data block is stored in its corresponding first cache area, including:
[0016] Determine the data blocks to be distributed to each second processor;
[0017] The data blocks to be distributed to each second processor are sequentially stored in the free slots of the circular queue on the remote direct memory access channel between the second processor and the first processor.
[0018] Optionally, the data communication method further includes:
[0019] Record the metadata information of each data block when it is stored in the corresponding first cache area.
[0020] Optionally, the target notification signal is further used to indicate the location of the target data block in the first cache area;
[0021] The method further includes:
[0022] Based on the target notification signal, release the target data block in the first cache area;
[0023] In response to the existence of remaining data blocks not stored in the first cache area, at least a portion of the remaining data blocks are stored in the first cache area to release the free slots formed after the target data block.
[0024] Optionally, the data communication method further includes:
[0025] Based on the target notification signal, update the first counting information and the second counting information;
[0026] Wherein, the first counting information represents the number of data blocks received by each of the second processors, and the second counting information represents the number of data blocks received by each processing component included in the corresponding part of the inference model deployed on each of the second processors.
[0027] A data communication method applied to a second processor, the method comprising:
[0028] Receive the target data block transmitted by the first processor through the corresponding direct memory access channel;
[0029] A target notification signal is sent to the first processor, the target notification signal indicating that the second processor has completed receiving the target data block;
[0030] Based on the target notification signal, model training or inference is performed on the target data block;
[0031] Receive the next data block of the target data block sent by the first processor based on the target notification signal;
[0032] Each data block includes at least one lexical element from the lexical sequence of the target data.
[0033] Optionally, the data communication method further includes:
[0034] The target data block is stored in the second cache area;
[0035] The target data block is obtained from the second cache area based on the target notification signal, so as to perform model training or inference on the target data block;
[0036] The second cache region is a pre-allocated cache region on the second processor, and the size of the second cache region satisfies the consistency condition with the maximum number of data blocks that the direct memory access channel can transmit at one time.
[0037] A first processor includes a first processing unit and a first interface;
[0038] The first processing unit is configured to perform the following processing:
[0039] Obtain the word sequence of the target data, and transform the word sequence into multiple data blocks; each data block includes at least one word from the word sequence.
[0040] Based on the data block distribution strategy, each data block is stored in a corresponding first cache area; each first cache area corresponds to a corresponding second processor, and each first cache area is located on the direct memory access channel between the corresponding second processor and the first processor; the direct memory access channel is connected to the first interface;
[0041] Transmit the target data block currently to be transmitted in the first cache region to the corresponding second processor;
[0042] The corresponding second processor receives a target notification signal for the target data block, the target notification signal indicating that the corresponding second processor has completed receiving the target data block;
[0043] The next data block of the target data block in the first cache region is transmitted to the corresponding second processor. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 This is a schematic diagram of the MoE model structure and communication method provided in this application;
[0046] Figure 2 This is a flowchart of a data communication method applied to a first processor provided in this application;
[0047] Figure 3 This is another flowchart of the data communication method applied to the first processor provided in this application;
[0048] Figure 4 This is yet another flowchart of the data communication method applied to a first processor provided in this application;
[0049] Figure 5 This is a flowchart of a data communication method applied to a second processor provided in this application;
[0050] Figure 6 This is a schematic diagram of the block-based pipelined transmission mechanism combined with a notification mechanism provided in this application;
[0051] Figure 7 This is a structural diagram of the first processor provided in this application;
[0052] Figure 8 This is a structural diagram of the second processor provided in this application. Detailed Implementation
[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0054] Hybrid expert models, or MOE models, have been widely used in large language models to scale up the parameters of these models to the trillions while maintaining a fixed computational cost. See also Figure 1The provided diagram illustrates the structure and communication method of the MoE model. In distributed training or inference scenarios, the MoE model is deployed on the entire distributed cluster. Specifically, one MoE model is deployed on a distributed cluster, and each node in the distributed cluster deploys a part of the MoE model, such as one or more experts from the various experts included in the MoE model on each node.
[0055] like Figure 1 The distributed cluster shown is illustrated, where each Worker represents a node within the cluster. Taking a GPU (Graphics Processing Unit) distributed cluster as an example... Figure 1 Each Worker can represent a GPU, and each GPU deploys one or more experts of the MOE model. For example, Worker1 deploys three experts, Expert1, Expert2, and Expert3, from the experts included in the MOE model, and Worker2 also deploys three experts, Expert4, Expert5, and Expert6, from the MOE model.
[0056] In the MOE model, an expert refers to multiple specialized sub-models contained within the model. Each sub-model is usually a neural network responsible for processing a specific part or pattern of the input data. Correspondingly, different nodes in the distributed cluster (such as different GPUs) are responsible for processing specific parts or patterns of the input data based on their respective deployed experts, and model training or inference is achieved through collaboration / interaction between nodes, data / pattern integration, etc.
[0057] In this application, the MOE model can also be referred to as a large language model implemented based on the MOE architecture. In distributed inference or training scenarios, MoE brings significant pain points in communication requirements, mainly manifested in the following ways:
[0058] 1) Complex communication mode: In traditional ALL-to-ALL communication, each token of the model input data must communicate directly point-to-point with the device where the target expert is located (such as a GPU in a cluster). When the token batch size is large and the number of experts is large, the communication traffic will explode.
[0059] 2) Severe network congestion: Because the mapping from token to expert is sparse and dynamic, the communication load is highly unbalanced among devices, which can easily cause hotspot nodes and link congestion.
[0060] 3) Significant latency bottleneck: The number of small data packets in ALL-to-ALL communication is huge, and frequent cross-node communication will become the performance bottleneck of the entire inference or training process, overshadowing the computational advantages of MoE.
[0061] Therefore, the core challenge of the MoE model in distributed scenarios has shifted from the bottleneck of computing power to the bottleneck of high-performance communication.
[0062] To address the communication challenges of MOE, embodiments of this application provide a data communication method and processor to meet the high communication requirements of MOE and improve the training or inference efficiency of models based on the MoE architecture (such as large language models).
[0063] The methods provided in this application include a data communication method applied to a first processor and a data communication method applied to a second processor. The first and second processors may be, but are not limited to, processors such as GPUs and NPUs (Neural-network Processing Units) capable of being used for AI (Artificial Intelligence) model training and inference.
[0064] Furthermore, the first processor and the second processor can be different processors in a distributed cluster, each deploying different parts of an inference model to perform model training or inference on the received data (such as data blocks) based on the corresponding parts of the deployed inference model. For example, the first processor and the second processor could be different GPUs in a GPU distributed cluster, each deploying different experts from the experts included in the MOE model, to perform model training or inference on the received data (such as data blocks) based on the deployed different experts.
[0065] See Figure 2 The flowchart shown illustrates the data communication method applied to a first processor provided in this application embodiment, which may include the following steps 201 to 205, which are described in detail below.
[0066] Step 201: Obtain the word sequence of the target data and convert the word sequence into multiple data blocks; each data block includes at least one word from the word sequence.
[0067] The target data refers to the input data to be processed by the inference model (MOE model / large language model based on MOE architecture) deployed in the distributed cluster in the application scenario. For example, it can be the sample data to be trained in the model training scenario, or the task data to be inferred in the model inference scenario, such as the task data of the user input task to be processed (such as the question to be answered).
[0068] Optionally, the target data can be input data in natural language form.
[0069] The target data's lexical sequence can be the lexical sequence formed from the token stream of the input data, including each token of the input data.
[0070] In this embodiment of the application, after obtaining the target data (such as the task data of the task to be processed input by the user), the first processor can first convert the target data into a word sequence such as a token sequence, and then perform segmentation processing on the word sequence of the target data according to the preset block size / block dimensions, so as to divide the word sequence of the target data into multiple data blocks. Each data block includes at least one word in the word sequence, which facilitates the subsequent distribution of each data block to each second processor in the distributed cluster for model training or inference.
[0071] Each data block includes at least one lexical element from the lexical sequence. This can refer to each data block including at least one original lexical element from the lexical sequence, or it can refer to each data block including at least one original lexical element from the lexical sequence of KV (Key-Value) data. There is no restriction on this, and it can be determined according to the actual application.
[0072] The first processor and the second processor can be regarded as the data block distribution node and the receiving node in the distributed cluster, respectively, or they can be called the sender and the receiver, respectively.
[0073] In practical applications, the first processor can specifically be one of the processors in the distributed cluster responsible for converting the target data into a word sequence, such as a token sequence. In this embodiment, the first processor, responsible for converting the target data into a word sequence, is further equipped with a data block segmentation and distribution function. This allows the first processor to further segment the word sequence into multiple data blocks after converting the target data into a word sequence and then distribute these data blocks. The second processors are the processors in the distributed cluster used for model training or inference based on the segmented data blocks. There are typically multiple second processors.
[0074] Step 202: Based on the data block distribution strategy, store each data block in its corresponding first cache area.
[0075] Each first cache region corresponds to a corresponding second processor, and each first cache region is located on the RDMA (Remote Direct Memory Access) channel between the corresponding second processor and the first processor.
[0076] In this embodiment, an RDMA channel is constructed between each second processor and the first processor, and different RDMA channels are constructed between different second processors and the first processor, for the first processor to transfer data blocks to different second processors respectively. Simultaneously, a first cache region is created on each RDMA channel between the second processor and the first processor, and different first cache regions created on different RDMA channels are used by the first processor to cache data blocks to be distributed to different second processors.
[0077] RDMA channels can be built on, but are not limited to, PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) buses.
[0078] The first buffer region created on the RDMA channel may, but is not limited to, be a circular queue.
[0079] After dividing the target data's lexical sequence into multiple data blocks, the first processor can store each data block into its corresponding first cache area based on a data block distribution strategy. Optionally, this process can be further implemented as follows: the first processor first determines the data block to be distributed to each second processor, and then stores the data block to be distributed to each second processor sequentially into an empty slot in the circular queue on the RDMA channel between the second processor and the first processor.
[0080] The different experts in a MOE model typically have different functions. This difference is one of the core design principles of hybrid expert models, which aims to improve overall performance through specialization. For example, each expert network will naturally develop expertise in specific types of data or task subdomains during training. For instance, when dealing with language modeling tasks, some experts may be better at handling grammatical structures, while others may perform better in understanding specific domain terms (such as medicine or law) or handling different language styles.
[0081] Based on this, the first processor may, but is not limited to, determine the data blocks to be distributed to each second processor based on the differences between different parts of the inference model deployed on different second processors, such as the differences in functionality and / or number of different experts in the MOE model deployed on different processors, and then cache the data blocks to be distributed to each second processor in a circular queue on the RDMA channel corresponding to that second processor.
[0082] Optionally, the first processor may also record metadata information of each data block when it is stored in the corresponding first cache area, such as recording metadata information of each data block when it is stored in the corresponding circular queue.
[0083] The recorded metadata information may include, but is not limited to, some or all of the information such as data block size, channel number, sequence number / slot number, etc., wherein the channel number is used to identify the RDMA channel corresponding to the first buffer area where the data block is located, such as the circular queue, and the sequence number / slot number is used to identify the slot corresponding to the data block in the first buffer area.
[0084] Step 203: Transmit the target data block currently to be transmitted in the first cache area to the corresponding second processor.
[0085] Based on step 202, the first processor may further transmit the target data block to be transmitted in the first cache area corresponding to each second processor according to a preset data transmission strategy, based on the data block cached in the first cache area constructed on each RDMA channel.
[0086] For example, according to the first-in-first-out data transmission strategy, the data block with the earliest current storage time in the circular queue on the RDMA channel corresponding to each second processor is taken as the target data block to be transmitted, and the target data block is transmitted to the corresponding second processor through the RDMA channel it is in.
[0087] Since the first buffer area, such as the circular queue, is created in the RDMA channel, when transmitting data blocks, the target data block to be transmitted can be directly transmitted from the circular queue through the RDMA channel and written to the second buffer area of the receiving end, such as the video memory of the second processor at the receiving end, without intermediate copying. This can significantly improve the data block transmission efficiency and correspondingly improve the communication efficiency between the data block and the device where its corresponding target expert is located (such as the GPU node in the cluster).
[0088] Step 204: Receive the target notification signal from the corresponding second processor for the target data block, wherein the target notification signal indicates that the corresponding second processor has completed receiving the target data block.
[0089] After the first processor transmits the target data block to each second processor through the corresponding RDMA channel based on the data blocks cached in each first cache region, each second processor can receive the target data block sent by the distribution node, i.e., the first processor, through the corresponding RDMA channel. After receiving the target data block, it can generate a target notification signal indicating that the target data block has been received, and then send the target notification signal to the first processor so that the first processor can know the data block reception status of each second processor in a timely manner.
[0090] After receiving the target data block and generating the target notification signal, the second processor can also perform model training or inference on the target data block based on the target notification signal, without waiting for the full data block to arrive before starting the processing of the data block, so as to realize low-latency pipelined processing of the data block by receiving and calculating at the same time (i.e., "calculate as soon as the block arrives").
[0091] Step 205: Transmit the next data block of the target data block in the first cache region to the corresponding second processor.
[0092] After receiving the target notification signal from the corresponding second processor, the first processor continues to transmit the next data block of the target data block in the corresponding first buffer area, such as in the circular queue, to the second processor through the RDMA channel corresponding to the second processor. This is done in conjunction with the receiving end's block-on-demand method to realize the block-by-block pipelined transmission of each data block in the first buffer area.
[0093] The first processor will transmit each data block in each first buffer area based on the above-mentioned block-based pipelined transmission mechanism until each data block in each first buffer area is transmitted to the corresponding second processor at the receiving end. The second processor at each receiving end will then continuously perform model training or inference on each received data block in a "compute as soon as block arrives" manner, without waiting for the full amount of data to arrive.
[0094] In summary, the data communication method provided in this application establishes RDMA channels between the first processor at the sending end and each of the second processors at the receiving end, and constructs a first buffer area on the RDMA channel between the first processor and each of the second processors. Based on this, the word sequence of the target data is converted into multiple data blocks. Each data block is stored in the corresponding first buffer area based on the data distribution strategy. Combined with a notification mechanism, the data blocks in each first buffer area are transmitted to the second processor at the receiving end through the corresponding RDMA channel in a block-based pipelined transmission manner, so that each second processor can perform model training or inference on the data blocks in a "compute as soon as the block arrives" manner.
[0095] It should be noted that the first processor can also perform model training or inference based on the corresponding part of its deployed inference model (such as the corresponding expert of the MOE model) based on data blocks. That is, the first processor can also have the requirement to receive data blocks. In this case, the first processor can directly store the data blocks to be distributed to itself in its own (first processor's) cache space (such as video memory) based on the data block distribution strategy.
[0096] The aforementioned data communication and processing mechanism, on the one hand, avoids intermediate copying of data blocks, significantly improving the data transmission efficiency between different processors in the distributed cluster. Furthermore, the combination of the notification mechanism and block transmission makes data transmission more refined and controllable, avoiding the impact of large-scale burst data transfer on cluster bandwidth, thus solving the communication demand pain points brought about by MOE. On the other hand, the second processor at the receiving end does not need to wait for all data blocks to be transmitted before starting processing, thereby improving the training or inference efficiency of models based on the MoE architecture (such as large language models) and significantly reducing the first-to-first-to-toe latency (TTFT) of user task requests.
[0097] In an optional embodiment, the target notification signal received by the first processor from the corresponding second processor is further used to indicate the location of the target data block in the corresponding first cache area. For example, the target notification signal may include, but is not limited to, the slot number / sequence number of the target data block in the corresponding first cache area, such as a circular queue, and the channel number of the RDMA channel in which the circular queue or other first cache area is located, so as to combine the channel number and the slot number / sequence number to indicate the location of the target data block in the corresponding first cache area.
[0098] See Figure 3 The flowchart shown illustrates that, in this embodiment, the data communication method applied to the first processor provided in this application may further include the following processes:
[0099] Step 206: Based on the target notification signal, release the target data block in the first cache area.
[0100] Once the first processor receives a target notification signal from a second processor, it can know that the second processor has completed receiving the target data block sent to it by the first processor. Based on this, the first processor can release the target data block in the corresponding first buffer area. For example, it can directly remove the target data block from the circular queue based on the channel number and slot number / sequence number of the target data block contained in the target notification signal, so that the slot occupied by the target data block changes from an occupied state to an idle state, and can then be used to store unbuffered data blocks to be transmitted.
[0101] Step 207: In response to the existence of remaining data blocks not stored in the first cache area, at least a portion of the remaining data blocks are stored in the first cache area to release the free slots formed after the target data block.
[0102] After the target data block is cleared from the corresponding first cache area, if there are still unbuffered data blocks waiting to be transmitted to the corresponding second processor on the first processor side of the RDMA channel corresponding to the first cache area, one of the remaining data blocks is stored in the free slot formed after the target data block is released in the first cache area. For example, one of the remaining data blocks is stored in the free slot formed after the target data block is released in the corresponding circular queue, so that the data blocks that have not been transmitted to the second processor can be transmitted to the second processor in sequence according to the block pipeline transmission mechanism based on the first-in-first-out strategy.
[0103] Meanwhile, for the remaining data blocks cached in the first cache area such as the circular queue, metadata information can also be recorded, such as the data block size, channel number, sequence number / slot number, etc. corresponding to the data block.
[0104] A circular queue is a first-in, first-out (FIFO) data structure that forms a closed loop by logically connecting the head and tail of the queue. Its core implementation relies on structure encapsulation and pointer operations. This structure uses front and rear pointers to manage data access and uses modulo operations to achieve circular access to memory addresses.
[0105] A typical linear queue uses contiguous blocks of memory (such as arrays). The head and tail pointers move linearly, and dequeueing requires shifting remaining elements forward, which can easily lead to memory waste and "false overflow" issues. In contrast, a circular queue treats memory as a circular structure. The tail pointer automatically loops back to the beginning after reaching the end, and uses modulo operations to achieve circular reuse. This avoids element shifting and "false overflow" problems, significantly improving cache space utilization.
[0106] This embodiment combines a notification mechanism with block-based pipelined transmission to achieve pipelined transmission of each data block to be transmitted to the receiving end. This makes data transmission between the sending and receiving ends more refined and controllable, avoiding the impact of large-scale burst data transfer on cluster bandwidth. By constructing a circular queue or other first buffer area on the RDMA channel between the sending and receiving ends, intermediate copying during data transmission is avoided, improving communication efficiency and thus solving the communication pain points brought about by MOE. In addition, by constructing a circular queue as the first buffer area, element movement and "false overflow" problems can also be avoided, significantly improving buffer space utilization and operational efficiency.
[0107] In an alternative embodiment, see Figure 4 The flowchart shown in this application illustrates that the data communication method applied to a first processor provided in this application may further include the following processes:
[0108] Step 208: Update the first counting information and the second counting information based on the target notification signal.
[0109] The first counting information represents the number of data blocks that each second processor has received, and the second counting information represents the number of data blocks that each processing component in the corresponding part of the inference model deployed on each second processor has received.
[0110] Optionally, the processing component can be an expert included in an inference model such as an MOE model. As mentioned above, in an MOE model, an expert refers to multiple specialized sub-models contained within the model. Each sub-model is typically a neural network responsible for processing a specific part or pattern of the input data.
[0111] In this embodiment, the target notification signal fed back by the second processor can also indicate which processing component, such as which expert, in the corresponding part of the inference model deployed on the second processor received the target data block that has been received, so that the first processor can update the second count information while updating the first count information based on the target notification signal.
[0112] This embodiment updates the first and second counting information based on the target notification signal, which makes it easier for the first processor to understand the receiving end's reception status of the data block in a timely manner. This facilitates the combination of a block-based pipelined transmission mechanism, making the data transmission between the sending end and the receiving end more refined and controllable.
[0113] See Figure 5 The flowchart shown illustrates the data communication method applied to a second processor provided in this application embodiment, which may include the following steps 501 to 504, which are described in detail below.
[0114] Step 501: Receive the target data block transmitted by the first processor through the corresponding direct memory access channel.
[0115] For example, in a distributed GPU cluster, the receiving GPU can receive target data blocks distributed by the sending GPU through an RDMA channel between itself and the sending GPU.
[0116] Optionally, after receiving the target data block, the second processor can store the target data block in the second cache area so that the target data block can be read from the second cache area for model training or inference later.
[0117] The second cache region is a pre-allocated cache region on the second processor. Taking the second processor as a GPU in a GPU distributed cluster as an example, the second cache region can be a pre-allocated area in the GPU's video memory for data block caching.
[0118] Optionally, the size of the second cache region satisfies the consistency condition with the maximum number of data blocks that can be transferred at one time through the RDMA channel between the second processor and the first processor.
[0119] Optionally, the consistency condition can be set to the size of the second cache region being the same as the maximum number of data blocks that can be transferred at one time by the RDMA channel between the second processor and the first processor, or the size of the second cache region being different from the maximum number of data blocks that can be transferred at one time by the RDMA channel between the second processor and the first processor, but the absolute value of the difference between the two is less than a threshold.
[0120] Based on the above consistency conditions and combined with the block-to-block computation mechanism of the second processor, the second cache area in the second processor only needs to cover the amount of data in transit (the amount of data of each data block being transmitted on the RAMA channel), thereby avoiding the large-scale one-time allocation of video memory by the second processor (such as the receiving GPU) for the full amount of data to be received, saving cache resource overhead on the second processor side.
[0121] Step 502: Send a target notification signal to the first processor, wherein the target notification signal indicates that the second processor has completed receiving the target data block.
[0122] After receiving the target data block, the second processor can generate a target notification signal indicating that it has completed receiving the target data block, and send the target notification signal to the first processor.
[0123] The target notification signal may include the slot number / sequence number of the target data block in the corresponding first cache area, such as a circular queue, and the channel number of the RDMA channel corresponding to the circular queue or other first cache area, to indicate the position of the target data block in the corresponding first cache area, so that the first processor can clear the target data block from the corresponding circular queue or other first cache area based on the target notification signal.
[0124] In addition, the target notification signal can also indicate which processing component (such as which expert) in the corresponding part of the inference model deployed on the second processor receives the target data block, so that the first processor can update the second count information while updating the first count information based on the target notification signal.
[0125] Step 503: Based on the target notification signal, perform model training or inference on the target data block.
[0126] After generating the target notification signal, the second processor can also read the target data block from the second cache area based on the target notification signal, and perform related processing such as model training or inference based on the target data block. For example, in the model training stage, the corresponding experts in the MOE model deployed on the second processor perform relevant model training on the target data block; in the model inference stage, the corresponding experts in the MOE model deployed on the second processor perform relevant model inference on the target data block.
[0127] Step 504: Receive the next data block of the target data block sent by the first processor based on the target notification signal.
[0128] Each data block includes at least one lexical element from the lexical sequence of the target data.
[0129] While the second processor performs model training or inference on the received target data block, it can also receive the next data block of the target data block transmitted by the first processor through the RDMA channel. Subsequently, the received next data block can be processed in the same way as the target data block. For example, the next data block can be stored in the second cache area of the second processor, a target notification signal representing the completion of receiving the next data block can be generated, and the signal can be fed back to the first processor. At the same time, based on the signal, the next data block can be read from the second cache area and model training or model inference can be performed on it. This allows the second processor to perform data communication and processing in a "receive and compute / compute as block arrives" mode, without having to wait for all data blocks to arrive before starting the processing of data blocks, until the second processor completes the reception and processing of all data blocks distributed by the first processor based on the "receive and compute / compute as block arrives" mode.
[0130] When the last data block to be distributed to the second processor is transmitted to the second processor for the second processor to perform the last batch of calculations, the second processor can generate a request to end signal and feed the signal back to the first processor. After receiving the request to end signal, the first processor clears the corresponding circular queue and releases the slot.
[0131] In summary, the data communication method provided in this embodiment, through the combination of a notification mechanism and block transmission, enables data transmission between the first and second processors, making data transmission more refined and controllable. This avoids the impact of large-scale burst data transfer on cluster bandwidth. Furthermore, by constructing a first buffer area, such as a circular queue, on the RDMA channel between the first and second processors to cache the data blocks to be transmitted to the second processor, intermediate copying of data blocks during data block transmission is effectively avoided, thus solving the communication requirement pain points brought about by MOE. In addition, this embodiment improves the training or inference efficiency of models (such as large language models) based on the MoE architecture by processing the data blocks distributed by the first processor on the receiving end in a "receive and compute / compute upon block arrival" mode on the second processor, and can significantly reduce the first-word latency of user task requests.
[0132] In an optional embodiment, the data communication method applied to a second processor provided in this application may further include the following processing before model training or inference is performed on the target data block:
[0133] a. In response to the target data block being of the target data type, the target data block is dequantized, and the dequantization result is decoded to perform model training or inference based on the decoding result.
[0134] The target data type indicates that the target data block is a quantized data block. The target data type may be, but is not limited to, the FP8 data type.
[0135] FP8 data is a low-precision representation of raw high-precision data (such as FP16 or BF16) after quantization. FP8 uses quantization techniques (such as linear quantization) to map high-precision data (such as FP16 or FP32) to an 8-bit representation, compressing the data by reducing the number of bits required to represent each value, thereby reducing storage and computational overhead. During this process, a scaling factor is used as a quantization parameter to adjust the numerical range, ensuring that the low-precision representation covers the dynamic range of the original data.
[0136] After receiving the target data block, the second processor can first determine its data type. If the target data block is of a target data type such as FP8 data, the second processor can perform dequantization processing on the target data block. For example, for the target data block of FP8 data type, scale recovery can be performed (based on the scaling factor, the quantized low-precision integer value is mapped back to the original high-precision floating-point range) to achieve precision conversion and restore the high-precision data corresponding to the target data block.
[0137] Based on this, the dequantization results can be further decoded to enable model training or inference.
[0138] b. In response to the target data block being a non-target data type, the target data block is decoded to perform model training or inference based on the decoding result;
[0139] If the target data block is a non-target data type such as non-FP8 data, indicating that the target data block is a non-quantized data block, the second processor can directly decode the target data block to perform model training or inference based on the decoding result.
[0140] In this embodiment, after receiving the target data block sent by the first processor, the second processor first determines the data type of the target data block, and if the target data block is of the target data type such as FP8, it performs dequantization processing on the target data block to restore the original high-precision data of the target data block, thereby ensuring the accuracy of model training or inference at the receiving end based on the target data block.
[0141] The following is an application example of the method of this application.
[0142] In this example, the first processor and the second processor are the sending GPU and the receiving GPU in the GPU distributed cluster, respectively, and can also be called the distribution node and the receiving node. The number of second processors, i.e., receiving GPUs, can be multiple. Different experts from the experts included in the MOE model can be deployed on each GPU node. An RDMA channel is built between the sending GPU and each receiving GPU, and a circular queue is established on each RDMA channel.
[0143] The workflow of the nodes in this GPU distributed cluster is as follows.
[0144] 1. Input task enters the distribution node, the distribution node performs data splitting and data block distribution.
[0145] 1) After receiving the input task, the distribution node converts the task data into a token stream (word sequence) and splits the token stream into multiple data blocks containing C tokens each. Each data block can specifically include KV data of C tokens.
[0146] Where C is an integer greater than or equal to 1.
[0147] 2) Allocate a slot in the circular queue on the corresponding RDMA channel for each data block, and record metadata such as channel number, sequence number, and data size. Each slot is identified by the channel number and slot number / sequence number.
[0148] 3) Write the vector data corresponding to each data block into a circular queue in the pre-allocated RDMA.
[0149] II. Data transmission between the distribution node and the receiving node
[0150] 1) After the transmitting end completes the filling (buffering) of the data block into the circular queue of each RDMA channel, it directly transmits the data block to be transmitted in the circular queue to the corresponding slot in the receiving end's video memory through the corresponding RDMA channel.
[0151] 2) Because of the use of block (data block) transmission, the receiving end can trigger model training or inference calculations on the data block when the first data block arrives, without waiting for all data blocks to be transmitted, thereby reducing latency and the first word delay of user task requests.
[0152] 3) After receiving each data block sent by the sender, the receiver sends a target notification signal (notify) to the sender to indicate that the data block has been received. The sender can then know that the data in the nth slot of a certain channel has arrived at the receiver and can release the data block cached at that position in the circular queue to write the next uncached data block to the circular queue; at the same time, it can update the rank count information and expert count information.
[0153] The rank count information includes the number of data blocks that each receiving node has received, and the expert count information includes the number of data blocks that each expert in each receiving node has received.
[0154] III. Processing Procedures of the Receiving Node
[0155] 1) The receiving end is resident in the kernel to monitor the notify signal to detect whether the data block of the corresponding slot of the corresponding channel is ready. When the data block of the corresponding slot of a certain channel is detected to be ready, the data block is immediately retrieved and the calculation such as model training or inference is triggered without waiting for the arrival of other data blocks, thus realizing the calculation as soon as the block arrives.
[0156] 2) If the data block uses FP8 precision, the receiver will scale and restore the token in the data block after extracting the data block before triggering decoding. Otherwise, if it is not FP8, it will be decoded directly.
[0157] 3) Once decoding is complete, model training or inference can be performed based on the decoding results. While decoding the target data block and training or inferring the model, the slot occupied by the data block in the circular queue is released, and the remaining data blocks to be cached can continue to be written to the slot in the circular queue to realize the circular reuse of the circular queue cache resources. At the same time, the next data block of the target data block in the circular queue is transmitted to the receiving end through the RDMA channel.
[0158] IV. Request Termination and Resource Recycling
[0159] 1) When the last data block is transmitted and decoding is triggered, the receiving end still completes the last batch of calculations according to the data block granularity and triggers a request to end signal;
[0160] 2) After the request ends and the signal arrives at the sending end, the sending end clears the corresponding circular queue and releases the slot based on the notify mechanism;
[0161] 3) The pre-allocated RDMA channels and circular queues are used cyclically throughout the inference process and do not expand linearly with the size of requests, thus ensuring the scalability of the system in multi-request concurrent scenarios.
[0162] The GPU distributed cluster working mechanism in this example has the following characteristics:
[0163] 1) Low latency: Since each data block is consumed as soon as it arrives at the receiver (calculation upon arrival of the block), the model training or inference at the receiver can begin calculation when the earliest data block arrives, avoiding waiting for all data blocks to be transmitted.
[0164] 2) High throughput: By using a circular queue design with a transit multiple of k (i.e., the size of the circular queue is k times the transit data transmitted on the RDMA channel), multiple data blocks can be kept flowing on the RDMA link at the same time, thereby keeping the RDMA channel continuously saturated and avoiding resource waste.
[0165] 3) Storage optimization: The size of the pre-allocated buffer (second cache area) at the receiving end can be related only to the "number of data blocks in transit" (data blocks being transmitted on the RDMA channel) rather than the maximum total number of tokens. This avoids large-scale one-time allocation of video memory at the receiving end based on the maximum total number of tokens, saving cache resource overhead at the receiving end.
[0166] 4) Notification-driven pipelining communication: such as Figure 6 As shown, while transmitting data blocks on the RDMA channel, the circular queue and related counters are updated in conjunction with the notification mechanism. The receiving end does not need to wait for the full data to arrive, but starts decoding calculation immediately when each data block arrives and is ready. This achieves low-latency pipelined processing and can significantly reduce the first word latency of user task requests.
[0167] This application embodiment also provides a first processor, the structure of which is as follows: Figure 7 As shown, it includes a first processing unit 11 and a first interface 12;
[0168] The first processing unit is configured to implement any of the data communication methods applied to the first processor as provided in the corresponding embodiments above.
[0169] An RDMA channel is established between the first processor and each of the second processors, and a first cache area such as a circular queue is established on each RDMA channel.
[0170] The first processor and each of the second processors are different processors in the distributed cluster; the different processors in the distributed cluster are respectively deployed with different parts of the inference model, so as to perform model training or inference on the received data blocks based on the corresponding parts of the deployed inference model.
[0171] The first interface is connected to the RDMA channel between the first processor and each of the second processors.
[0172] This application also provides a second processor, the structure of which is as follows: Figure 8 As shown, it includes a second processing unit 11 and a second interface 12;
[0173] The second processing unit is used to implement any of the data communication methods applied to the second processor as provided in the corresponding embodiments above.
[0174] An RDMA channel is established between the second processor and the first processor, and a first cache area such as a circular queue is established on the RDMA channel.
[0175] The second processor and the first processor are different processors in the distributed cluster; the different processors in the distributed cluster are respectively deployed with different parts of the inference model, so as to perform model training or inference on the received data blocks based on the corresponding parts of the deployed inference model.
[0176] The second interface is connected to the RDMA channel between the second processor and the first processor.
[0177] In summary, compared with known technologies, the present application has at least the following technical advantages:
[0178] 1. Reduce request latency: By combining the notification mechanism with chunked transmission, computation can start as soon as the first data block arrives at the receiving end, avoiding waiting for the full token and significantly reducing the latency of the first token for inference.
[0179] 2. Improve system throughput: Block pipelining and circular queue mechanisms can maintain multiple data blocks in transit simultaneously to keep the RDMA channel continuously saturated, thereby improving the parallelism of cross-node communication and decoding and supporting larger batch processing scales.
[0180] 3. Reduce memory pressure: The pre-allocation of memory on the receiving end is changed from "full token" to "cache space required for the number of data blocks in transit", so that the system can still run stably in long text and high concurrency scenarios, avoiding the problem of memory explosion.
[0181] 4. Reduce cluster network bottleneck risk: The combination of notification mechanism and chunked transmission makes data transmission more refined and controllable, avoiding the impact of large-scale burst data transfer on cluster bandwidth, thereby improving the overall robustness and scalability of the cluster.
[0182] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0183] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.
[0184] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0185] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0186] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data communication method applied to a first processor, the method comprising: Obtain the word sequence of the target data and convert the word sequence into multiple data blocks; Each data block includes at least one lexical element from the lexical sequence; Based on the data block distribution strategy, each data block is stored in its corresponding first cache area; each first cache area corresponds to a corresponding second processor, and each first cache area is located on the direct memory access channel between the corresponding second processor and the first processor; Transmit the target data block currently to be transmitted in the first cache region to the corresponding second processor; The corresponding second processor receives a target notification signal for the target data block, the target notification signal indicating that the corresponding second processor has completed receiving the target data block; The next data block of the target data block in the first cache region is transmitted to the corresponding second processor.
2. The data communication method according to claim 1, wherein the first processor and the second processor are different processors in a distributed cluster; the different processors in the distributed cluster are respectively deployed with different parts of the inference model, so as to perform model training or inference on the received data blocks based on the corresponding parts of the deployed inference model.
3. The data communication method according to claim 1, wherein the number of second processors is plurality of; Different direct memory access channels are established between different second processors and the first processor; different circular queues are established on the direct memory access channels corresponding to different second processors. The first cache area includes the circular queue.
4. The data communication method according to claim 3, based on a data block distribution strategy, stores each data block separately in a corresponding first cache area, including: Determine the data blocks to be distributed to each second processor; The data blocks to be distributed to each second processor are sequentially stored in the free slots of the circular queue on the remote direct memory access channel between the second processor and the first processor.
5. The data communication method according to claim 1, further comprising: Record the metadata information of each data block when it is stored in the corresponding first cache area.
6. The data communication method according to claim 1, wherein the target notification signal is further used to indicate the location of the target data block in the first cache area; The method further includes: Based on the target notification signal, release the target data block in the first cache area; In response to the existence of remaining data blocks not stored in the first cache area, at least a portion of the remaining data blocks are stored in the first cache area to release the free slots formed after the target data block.
7. The data communication method according to claim 2 further includes: Based on the target notification signal, update the first counting information and the second counting information; Wherein, the first counting information represents the number of data blocks received by each of the second processors, and the second counting information represents the number of data blocks received by each processing component included in the corresponding part of the inference model deployed on each of the second processors.
8. A data communication method applied to a second processor, the method comprising: Receive the target data block transmitted by the first processor through the corresponding direct memory access channel; A target notification signal is sent to the first processor, the target notification signal indicating that the second processor has completed receiving the target data block; Based on the target notification signal, model training or inference is performed on the target data block; Receive the next data block of the target data block sent by the first processor based on the target notification signal; Each data block includes at least one lexical from the lexical sequence of the target data.
9. The data communication method according to claim 8, further comprising: The target data block is stored in the second cache area; The target data block is obtained from the second cache area based on the target notification signal, so as to perform model training or inference on the target data block; The second cache region is a pre-allocated cache region on the second processor, and the size of the second cache region satisfies the consistency condition with the maximum number of data blocks that the direct memory access channel can transmit at one time.
10. A first processor, comprising a first processing unit and a first interface; The first processing unit is configured to perform the following processing: Obtain the word sequence of the target data, and transform the word sequence into multiple data blocks; each data block includes at least one word from the word sequence. Based on the data block distribution strategy, each data block is stored in a corresponding first cache area; each first cache area corresponds to a corresponding second processor, and each first cache area is located on the direct memory access channel between the corresponding second processor and the first processor; the direct memory access channel is connected to the first interface; Transmit the target data block currently to be transmitted in the first cache region to the corresponding second processor; The corresponding second processor receives a target notification signal for the target data block, the target notification signal indicating that the corresponding second processor has completed receiving the target data block; The next data block of the target data block in the first cache region is transmitted to the corresponding second processor.