Communication and data processing method, device, equipment, medium and product
By offloading cross-node communication tasks from the CPU and using RDMA technology for cross-node communication, the problem of uneven GPU resource utilization is solved, and the overall throughput and training speed are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2025-12-09
- Publication Date
- 2026-05-05
AI Technical Summary
In existing technologies, GPUs cause uneven resource utilization and affect overall throughput when performing expert calculations and communication processing of token data.
By performing cross-node communication computations and data transfers on the CPU, the hidden state data of cross-node tokens is offloaded from the GPU to the CPU. The CPU is used for communication computations and cross-node communication tasks, thereby allowing the GPU to focus on expert computations or intra-node communication tasks. RDMA technology is used for cross-node communication to reduce latency.
It achieves balanced resource utilization, improves overall throughput, reduces communication latency, and enhances inference or training speed.
Smart Images
Figure CN121979653A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of cloud computing, big data models, and computing power, and specifically to a communication and data processing method, apparatus, device, medium, and product. Background Technology
[0002] To improve the performance of Large Language Models (LLMs), a Mixture of Experts (MoE) network can be introduced into the LLM. Summary of the Invention
[0003] This disclosure provides a communication and data processing method, apparatus, device, medium, and product.
[0004] According to one aspect of this disclosure, a communication method is provided, applied to a CPU, the method comprising: receiving a target expert index and data sent by a current GPU; the target expert index being determined by the current GPU based on a token sequence, the data including: hidden state data of all tokens in the token sequence; storing the data in the memory of the CPU; performing a communication calculation process based on the target expert index to obtain communication metadata; the communication metadata including: target storage information and cross-node GPU information, the target storage information being storage information of the hidden state data of the cross-node tokens in the memory, the target experts of the cross-node tokens including cross-node experts, the cross-node experts being deployed on the cross-node GPUs; and sending the hidden state data of the cross-node tokens stored in the memory to the cross-node GPUs based on the communication metadata.
[0005] According to another aspect of this disclosure, a data processing method is provided, applied to a current GPU, the method comprising: receiving a token sequence, the token sequence including hidden state data of multiple tokens; determining a target expert index corresponding to the token sequence; sending the target expert index and data to a CPU, so that the CPU performs a communication calculation process based on the target expert index, and sends the hidden state data of cross-node tokens stored in memory to the cross-node GPU according to communication metadata; wherein, the data includes: hidden state data of all tokens in the token sequence; the target expert of the cross-node token includes a cross-node expert, the cross-node expert being deployed on the cross-node GPU.
[0006] According to another aspect of this disclosure, a communication device is provided, applied to a CPU, the device comprising: a receiving module, configured to receive a target expert index and data sent by a current GPU; the target expert index is determined by the current GPU based on a token sequence, and the data includes: hidden state data of all tokens in the token sequence; a storage module, configured to store the data in the memory of the CPU; a calculation module, configured to perform a communication calculation process according to the target expert index to obtain communication metadata; the communication metadata includes: target storage information and cross-node GPU information, the target storage information being the storage information of the hidden state data of the cross-node tokens in the memory, the target experts of the cross-node tokens including cross-node experts, and the cross-node experts being deployed on the cross-node GPUs; and a communication module, configured to send the hidden state data of the cross-node tokens stored in the memory to the cross-node GPUs according to the communication metadata.
[0007] According to another aspect of this disclosure, a data processing apparatus is provided, applied to a current GPU, the apparatus comprising: a receiving module for receiving a token sequence, the token sequence including hidden state data of multiple tokens; a determining module for determining a target expert index corresponding to the token sequence; and a sending module for sending the target expert index and data to a CPU, so that the CPU performs a communication calculation process based on the target expert index, and sends the hidden state data of cross-node tokens stored in memory to the cross-node GPU according to communication metadata; wherein the data includes: hidden state data of all tokens in the token sequence; the target expert of the cross-node token includes a cross-node expert, the cross-node expert being deployed on the cross-node GPU.
[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.
[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.
[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.
[0011] According to embodiments of this disclosure, resource utilization can be balanced and overall throughput can be improved.
[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0014] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0015] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;
[0016] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;
[0017] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0018] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0019] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;
[0020] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure;
[0021] Figure 8 This is a schematic diagram of an electronic device used to implement the communication method or data processing method of the embodiments of this disclosure. Detailed Implementation
[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0023] To better understand the embodiments of this disclosure, the relevant terms are explained as follows:
[0024] Graphics Processing Unit (GPU): A microprocessor specifically designed for processing graphics and image-related computations. GPUs play a crucial role in the field of Artificial Intelligence (AI). Deep learning algorithms (such as neural networks) involve a large number of matrix operations, and the parallel computing capabilities of GPUs are well-suited for handling these operations. Therefore, GPUs are often used as the computing unit in AI scenarios.
[0025] A GPU processes all the data in a batch at once. The processing unit is usually called a token, which can be a text unit or an image unit, etc.
[0026] Specifically, token data can be the token's hidden states, which are obtained by extracting and processing features from the token's input data (such as characters or words).
[0027] MoE Network: In the MoE architecture, each MoE network includes a first number (e.g., 256) of candidate experts. After the GPU receives the token sequence, for each token in the sequence, a gating network is used to determine a second number (e.g., 8) of target experts from the first number of candidate experts. The second number of target experts are then used to perform expert computation on the hidden state data of the token.
[0028] The target experts mentioned above can include: local experts, experts on the same node, and cross-node experts. Local experts are deployed on the current GPU. Experts on the same node are deployed on the same node GPU, which is different from the current GPU but located on the same node (such as a server). Experts on cross-nodes are deployed on cross-node GPUs, which belong to different nodes than the current GPU.
[0029] For a given token, if the target expert includes a local expert, then the local expert performs expert computation on the token's hidden state data on the current GPU. If the target expert includes an expert from the local node, then the token's hidden state data is sent to the local node's GPU via intra-node communication, and the local node's GPU performs expert computation on the token's hidden state data. If the target expert includes a cross-node expert, then the token's hidden state data is sent to the cross-node GPU via cross-node communication, and the cross-node GPU performs expert computation on the token's hidden state data.
[0030] The aforementioned cross-node communication can specifically be RDMA communication.
[0031] Remote Direct Memory Access (RDMA): A direct memory access technology that transfers data directly from the memory of one computer to the memory of another computer.
[0032] RNIC (RDMA-enabled Network Interface Controller): A network interface card that supports RDMA technology and enables RDMA-based network transmission functions.
[0033] Central Processing Unit (CPU): The core computing and control unit of a computer system.
[0034] In related technologies, expert calculations of token data and communication-related processing are all performed by the GPU, while hardware such as the CPU is relatively idle. This can lead to problems such as uneven resource utilization and impact on overall throughput.
[0035] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure, which provides a communication method. This method is applied to a CPU, such as... Figure 1 As shown, the method includes:
[0036] 101. Receive the target expert index and data sent by the current GPU; the target expert index is determined by the current GPU based on the token sequence, and the data includes: the hidden state data of all tokens in the token sequence.
[0037] 102. The data is stored in the memory of the CPU.
[0038] 103. Based on the target expert index, perform a communication calculation process to obtain communication metadata; the communication metadata includes: target storage information and cross-node GPU information, the target storage information is the storage information of the hidden state data of the cross-node token in the memory, the target expert of the cross-node token includes the cross-node expert, and the cross-node expert is deployed on the cross-node GPU.
[0039] 104. Based on the communication metadata, send the hidden state data of the cross-node token stored in the memory to the cross-node GPU.
[0040] The method in this embodiment can be executed by the CPU.
[0041] The current GPU refers to the GPU that belongs to the same node as the CPU. A node is, for example, a server, which can include one or more GPUs, and any one of these GPUs can be designated as the current GPU.
[0042] The token sequence currently received by the GPU includes data for multiple tokens, specifically the hidden states of each token. After receiving the token sequence, the GPU performs batch processing on the hidden states of all tokens within that sequence.
[0043] Specifically, for each token in the token sequence, the target expert corresponding to that token is calculated, thereby constructing a target expert index, which can record the correspondence between all tokens and target experts.
[0044] After the GPU calculates the target expert index, it sends the target expert index and data (the hidden state data of all tokens in the token sequence) to the CPU.
[0045] After receiving the data, the CPU stores it in its memory. Additionally, it can store the target expert index for later use.
[0046] For the target expert index, the CPU performs a communication calculation process based on the target expert index to obtain communication metadata.
[0047] Communication metadata refers to communication-related data, such as the source and destination addresses of the data to be transmitted.
[0048] In this embodiment, the communication metadata specifically includes: target storage information and cross-node GPU information.
[0049] Target storage information refers to the memory storage information of the hidden state data of cross-node tokens. This stored information may include, for example, the address, length, and access key.
[0050] During the communication computation process, the cross-node token and its corresponding cross-node GPU can be determined. After the CPU stores the data in memory, it can obtain the storage information of all tokens. After determining the cross-node token, the storage information corresponding to the cross-node token can be filtered out as the target storage information.
[0051] Cross-node tokens refer to tokens that need to be distributed across multiple nodes.
[0052] That is, the target experts corresponding to the cross-node token include cross-node experts, and the cross-node experts are deployed on cross-node GPUs.
[0053] Cross-node GPUs refer to GPUs located on different nodes than the current GPU.
[0054] For example, the current GPU is GPU1, and the token sequence includes token1, token2, and token3. Assume that the target expert corresponding to token1 is EP1, the target expert corresponding to token2 is EP2, and the target expert corresponding to token3 is EP3. EP1 is deployed on GPU1, EP2 on GPU2, and EP3 on GPU3. GPU1 and GPU2 are located on the same node, while GPU1 and GPU3 are located on different nodes. Then:
[0055] Cross-node token is token3, cross-node expert is EP3, and cross-node GPU is GPU3.
[0056] After obtaining the communication metadata, the CPU can also synchronize it with each GPU in the GPU cluster.
[0057] The GPU cluster is used to execute the current processing procedure, such as inference or training.
[0058] The GPUs in this GPU cluster include the current GPU, the local node GPU, and cross-node GPUs. The local node GPU is different from the current GPU, but it is located on the same node as the current GPU. The cross-node GPU is different from the current GPU and is located on a different node than the current GPU.
[0059] In this way, each GPU can obtain the communication metadata so that subsequent normal communication and processing can be carried out.
[0060] The synchronization process can specifically employ an all-to-all communication method. All-to-all communication is a core communication mode in distributed parallel computing, meaning that each node in the cluster needs to send data to all other nodes while simultaneously receiving data from all other nodes. It is a key method for achieving global data exchange and is widely used in scenarios such as large model training and high-performance computing.
[0061] Furthermore, the aforementioned communication calculation process and communication metadata synchronization process can be implemented using existing functions. For example, `get_dispatch_layout()` can be used for communication calculation, and `notify_dispatch()` can be used for communication metadata synchronization. Here, `get_dispatch_layout()` is a communication calculation function, and `notify_dispatch()` is a communication metadata synchronization function.
[0062] After the CPU calculates the communication metadata and synchronizes it to each GPU, it sends the hidden state data of the cross-node token stored in the memory to the cross-node GPU based on the communication metadata, thereby realizing cross-node communication for the cross-node token.
[0063] In this embodiment, the CPU receives the target expert index and data sent by the current GPU, performs the communication calculation process based on the target expert index, and performs cross-node communication based on the communication metadata. This can offload the communication calculation task and the cross-node communication task from the GPU to the CPU, allowing the GPU to focus on expert calculation tasks or intra-node communication tasks, thereby balancing the resource utilization of each hardware and improving the overall throughput. It can also make full use of idle CPU resources and reduce costs.
[0064] In some embodiments, sending the hidden state data of the cross-node token stored in memory to the cross-node GPU based on the communication metadata includes:
[0065] Create a task unit, which contains the target storage information and the cross-node GPU information;
[0066] In response to the task unit meeting preset conditions, a trigger command is sent to the network card, so that the network card obtains the task unit based on the trigger command, and sends the hidden state data of the cross-node token to the cross-node GPU according to the task unit.
[0067] After the CPU calculates and obtains the communication metadata, it can create a task unit based on the communication metadata. The task unit is used to provide relevant information about the communication task. For example, the task unit contains the target storage information and the cross-node GPU information.
[0068] When a task unit meets a preset condition (such as the number of task units reaching a preset number), the CPU sends a trigger command to the network card. This trigger command may include the identifier of the task unit.
[0069] When the network card receives the trigger command, it can obtain the corresponding task unit according to the task unit identifier, parse relevant information from the task unit, such as target storage information and cross-node GPU information, then obtain the hidden state data of the cross-node token from memory according to the target storage information, and send the hidden state data of the cross-node token to the cross-node GPU according to the cross-node GPU information.
[0070] The aforementioned triggering instructions can be implemented through a doorbell mechanism, which is a triggering mechanism that notifies the network card's processing task unit by writing information to the network card's doorbell register.
[0071] In this embodiment, by sending a trigger command to the network card when the task unit meets the preset conditions, the network card can be triggered to perform batch communication processing, thereby reducing resource overhead and improving communication efficiency.
[0072] In some embodiments, the target expert index and the data sent by the current GPU can be received via a first PCIe bus.
[0073] In some embodiments, trigger commands can be sent to the network card via a second PCIe bus.
[0074] Currently, GPUs, CPUs, and network cards can communicate with each other via the PCIe bus. Peripheral Component Interconnect Express (PCIe) is a high-speed serial computer expansion bus standard.
[0075] The PCIe bus between the GPU and CPU is called the first PCIe bus, and the PCIe bus between the CPU and the network card is called the second PCIe bus. Based on this:
[0076] The CPU receives the target expert index and the data sent by the current GPU through the first PCIe bus, and sends a trigger command to the network card through the second PCIe bus.
[0077] In addition, after the network card obtains the task unit based on the trigger instruction, it can parse the target storage information from the task unit. Then, the network card can also obtain the hidden state data of the cross-node token corresponding to the target storage information from the CPU's memory through the second PCIe bus, and then send the hidden state data of the cross-node token to the cross-node GPU.
[0078] In this embodiment, the GPU, CPU, and network card communicate via the PCIe bus, which enables simple and efficient communication between them.
[0079] In some embodiments, the network interface card (NIC) is an RNIC; the hidden state data of the cross-node token is sent by the RNIC to the cross-node GPU based on RDMA communication.
[0080] After the network card obtains the task unit, it can parse the target storage information and cross-node GPU information. Then, based on the target storage information, it can retrieve the hidden state data of the cross-node token from memory.
[0081] When the network card is an RNIC, RDMA can be used to directly access the video memory of the cross-node GPU.
[0082] The information for cross-node GPUs can include: the address of the cross-node GPU, and cross-node storage information. Cross-node storage information refers to the storage information of the data to be written in the video memory of the cross-node GPU. The storage information includes, for example, the address, length, access key, etc.
[0083] Based on this, after RNIC obtains the hidden state data of the cross-node token, it can directly write the hidden state data of the cross-node token into the video memory of the cross-node GPU based on the address of the cross-node GPU and the cross-node storage information.
[0084] In this embodiment, cross-node communication based on RDAM technology can further reduce communication latency and improve inference or training speed.
[0085] In some embodiments, the CPU includes multiple thread pools; the step of performing communication calculations based on the target expert index includes: dividing the target expert index into multiple groups; and using each thread pool to perform the communication calculations in parallel on each group of target expert indexes.
[0086] Multiple thread pools can be pre-configured on the CPU, and each thread pool includes one or more threads.
[0087] After receiving the target expert index, the CPU can group the target expert index according to preset rules.
[0088] The number of groups in the target expert index is the same as the number of thread pools. For example, if the number of thread pools is N (a positive integer greater than 1), then the target expert index is divided into N groups.
[0089] For example, an even distribution method can be used. Specifically, if there are 10 tokens in the token sequence and 2 threads in the thread pool, then the target expert indexes corresponding to 5 tokens can be grouped together, and each thread pool can process the target expert indexes corresponding to 5 tokens.
[0090] Based on this, assuming the two thread pools are referred to as the first thread pool and the second thread pool, and the two sets of target expert indexes are referred to as the first set of target expert indexes and the second set of target expert indexes, then:
[0091] The first thread pool calculates the communication metadata corresponding to the first set of target expert indexes based on the first set of target expert indexes to obtain the first communication metadata; the second thread pool calculates the communication metadata corresponding to the second set of target expert indexes based on the second set of target expert indexes to obtain the second communication metadata. The set of the first communication metadata and the second communication metadata is used as the final communication metadata, and the calculation process of the first communication metadata and the second communication metadata is executed in parallel.
[0092] In this embodiment, processing efficiency can be improved based on the above-described parallel execution process.
[0093] Figure 2 This is a schematic diagram based on a second embodiment of the present disclosure, in which a node is provided.
[0094] like Figure 2 As shown, node 200 includes: the current GPU 201, CPU 202, and network card. Figure 2 Taking the RNIC203 network card as an example, the network card used in China and Israel is RNIC203.
[0095] The current GPU 201 is used to store the hidden state data of all tokens in the received token sequence into the video memory, and to calculate the target expert index corresponding to the token sequence, and send the target expert index and data (the hidden state data of all tokens in the token sequence) to the CPU.
[0096] CPU 202 is used to store the data in memory; and to perform a communication calculation process according to the target expert index to obtain communication metadata, create a task unit according to the communication metadata, and send a trigger command to the network interface card (RNIC) when the task unit meets preset conditions.
[0097] RNIC 203 is configured to, in response to the trigger instruction, acquire the task unit, acquire the hidden state data of the cross-node token from the memory based on the target storage information in the task unit, and send the hidden state data of the cross-node token to the cross-node GPU based on the cross-node GPU information in the task unit.
[0098] The target storage information is the storage information of the hidden state data of the cross-node token in the memory, and the target expert of the cross-node token includes the cross-node expert, which is deployed on the cross-node GPU.
[0099] Currently, GPU 201 and CPU 202 communicate via the first PCIe bus, while CPU 202 and RNIC 203 communicate via the second PCIe bus.
[0100] like Figure 2 As shown, the current GPU 201 may include: video memory and computing units.
[0101] Video memory is used to store the hidden state data of all tokens in the received token sequence. Specifically, it can be HBM, where High Bandwidth Memory (HBM) is a high-performance storage device.
[0102] The computation unit is used to calculate the target expert index corresponding to the token sequence; and, when the token sequence contains a local token, to perform expert computation on the hidden state data of the local token.
[0103] In this context, a local token refers to a token of a target expert, including a locally deployed expert. For example, if a token sequence contains token1, and the target expert of token1 is EP1, and EP1 is deployed on the current GPU, then the GPU computing unit of the current GPU can use EP1 to perform expert calculations on the hidden state data of token1.
[0104] Additionally, current GPUs may include an intra-node communication unit (not shown in the diagram), which performs intra-node communication. Specifically, it sends the hidden state data of the local node's tokens within the token sequence to the local node's GPU.
[0105] For example, if the token sequence contains token2, the target expert of token2 includes EP2, EP2 is deployed on GPU2, the current GPU is GPU1, and GPU1 and GPU2 are located in the same node, then GPU2 is the local GPU, and token2 is the local token. Afterwards, GPU1 sends the hidden state data of token2 to GPU2 through the intra-node communication unit.
[0106] CPU 202 can perform communication calculations and obtain communication metadata. Then, it can create task units based on the communication metadata.
[0107] After receiving the target expert index through the first PCIe bus, the CPU performs communication calculations based on the target expert index to obtain communication metadata, such as the cross-node GPU information corresponding to the cross-node token, the target storage information corresponding to the cross-node token, etc. The target storage information refers to the storage information of the hidden state data of the cross-node token in memory.
[0108] After obtaining the target storage information and cross-node GPU information, the CPU records them in the task unit.
[0109] The CPU can also write the task unit into the memory so that the network card can read the task unit from the memory.
[0110] For example, after the CPU creates a task unit, it writes it into a queue in memory, and then the network card can retrieve the task unit from that queue.
[0111] In addition, once the number of task units reaches the preset number, the CPU sends a trigger command to the RNIC via the second PCIe bus. This trigger command may include the identifier of the task unit.
[0112] After receiving the trigger command, RNIC parses the identifier of the task unit, retrieves the corresponding task unit from memory based on the identifier, and obtains the target storage information and cross-node GPU information from the task unit.
[0113] Subsequently, RNIC retrieves the hidden state data of the cross-node token from memory based on the target storage information via the second PCIe bus, and writes the hidden state data of the cross-node token into the video memory of the cross-node GPU based on the cross-node GPU information using RDMA communication.
[0114] Based on the above process, the GPU, CPU, and RNIC communicate via the PCIe bus, offloading communication-related processing from the GPU to the CPU, thereby balancing hardware resources and improving overall throughput. Since communication-related processing is offloaded to the CPU, the GPU can use more resources for expert computation, thus processing more user requests per unit time. In addition, when the GPU performs cross-node communication, it usually needs to copy the hidden state data of the token corresponding to the same cross-node GPU to the GPU's contiguous buffer. However, in this embodiment, through the above-mentioned communication offloading, the contiguous buffer is not required in the GPU, thereby saving GPU memory resources.
[0115] Combination Figure 2 The architecture shown above, and the interaction process between the three types of hardware, can be found in [reference needed]. Figure 3 .
[0116] Figure 3 This is a schematic diagram based on a third embodiment of the present disclosure. This embodiment provides a data processing method, such as... Figure 3 As shown, the method includes:
[0117] 301. The current GPU receives a token sequence, which includes the hidden state data of multiple tokens. Calculate the target expert index corresponding to the token sequence.
[0118] 302. The current GPU sends the target expert index and data to the CPU, the data including: the hidden state data of all tokens in the token sequence.
[0119] 303. CPU, calculates communication metadata based on the target expert index, and stores the data in memory.
[0120] 304. CPU: Creates task units based on communication metadata.
[0121] 305. The CPU sends the task unit to the RNIC.
[0122] 306. RNIC obtains target storage information and cross-node GPU information from the task unit, and retrieves the hidden state data of the cross-node token from memory based on the target storage information.
[0123] 307. RNIC, using RDMA communication, writes the hidden state data of the cross-node token directly into the video memory of the cross-node GPU based on the information of the cross-node GPU.
[0124] For example, combining Figure 2 After receiving the token sequence, the GPU stores the hidden state data of all tokens in the token sequence in the video memory. Then, using a computing unit, for each token, a gating network is used to calculate the target expert of the token, establish the mapping relationship between the token and the target expert, and form the target expert index corresponding to the token sequence by combining the mapping relationships of all tokens in the token sequence.
[0125] Then, the current GPU sends the target expert index and data (hidden state data of all tokens in the token sequence) to the CPU via the first PCIe bus.
[0126] After receiving the target expert index, the CPU performs a communication calculation process based on the target expert index to obtain communication metadata.
[0127] The communication metadata includes: target storage information and cross-node GPU information.
[0128] Among them, the target storage information is the storage information of the cross-node token in memory.
[0129] After storing the data in memory, the CPU can obtain the storage information of all tokens. Based on the communication calculation process, it can determine the cross-node token in the token sequence. Then, the CPU can filter the storage information of the target token from the storage information of all tokens and use it as the target storage information.
[0130] Information about GPUs across nodes can also be obtained through the communication computing process.
[0131] After obtaining the target storage information and the information from the cross-node GPUs, the CPU writes it into the task unit and stores it in memory.
[0132] Once the task unit meets the preset conditions, such as reaching a preset number, the CPU sends a trigger command to the RNIC via the second PCIe bus, which carries the identifier of the task unit.
[0133] After receiving the trigger command, RNIC retrieves the corresponding task unit from memory based on the identifier in the command, and then retrieves the target storage information and cross-node GPU information from the task unit.
[0134] RNIC retrieves the hidden state data of cross-node tokens from the CPU's memory based on the target storage information via the second PCIe bus.
[0135] Afterwards, RNIC uses RDMA communication to directly write the hidden state data of the cross-node token into the video memory of the cross-node GPU.
[0136] In this embodiment, the CPU calculates communication metadata based on the target expert index sent by the GPU, and creates task units and sends trigger instructions. This allows communication calculation tasks and cross-node communication-related tasks to be offloaded from the GPU to the CPU, enabling the GPU to focus on expert calculations or intra-node communication tasks. This balances the resource utilization of each hardware component and improves overall throughput. Cross-node communication based on RDAM technology can further reduce communication latency and improve inference or training speed.
[0137] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides a data processing method, such as... Figure 4 As shown, the method includes:
[0138] 401. The current GPU receives the token sequence and calculates the target expert index corresponding to the token sequence.
[0139] 402. The current GPU sends the target expert index and data to the CPU, the data including: the hidden state data of all tokens in the token sequence.
[0140] 403. CPU: Calculates communication metadata based on the target expert index.
[0141] 404. CPU sends communication metadata to the current GPU.
[0142] 405. Currently, the GPU determines the local token and the token of this node in the token sequence based on the communication metadata.
[0143] 406. The current GPU retrieves the hidden state data of the local token from the video memory and performs expert calculations on the hidden state data of the local token.
[0144] 407. The current GPU retrieves the hidden state data of the local node token from the video memory and sends it to the local node GPU through intra-node communication.
[0145] 408. The GPU of this node performs expert calculations on the hidden state data of the token of this node.
[0146] After the CPU calculates and obtains the communication metadata, it can also synchronize it to each GPU through all-to-all communication, so that the current GPU can obtain the communication metadata.
[0147] Once the GPU obtains the communication metadata, it can determine the local token and the token of this node based on the communication metadata.
[0148] Among them, the target expert corresponding to the local token includes the local expert, which is deployed on the current GPU; the target expert corresponding to the node token includes the node expert, which is deployed on the node GPU. The node GPU is different from the current GPU, but it is located within the same node.
[0149] For example, the current GPU is GPU1, and the token sequence includes token1, token2, and token3. Assume that the target expert corresponding to token1 is EP1, the target expert corresponding to token2 is EP2, and the target expert corresponding to token3 is EP3. EP1 is deployed on GPU1, EP2 on GPU2, and EP3 on GPU3. GPU1 and GPU2 are located on the same node, while GPU1 and GPU3 are located on different nodes. Then:
[0150] Cross-node token is token3, cross-node expert is EP3, and cross-node GPU is GPU3;
[0151] The local token is token1, and the local expert's token is EP1.
[0152] The token for this node is token2, the expert for this node is EP2, and the GPU for this node is GPU2.
[0153] For a local token, the current GPU uses a corresponding local expert to perform expert calculations on its data. For example, GPU1 uses EP1 to perform expert calculations on the hidden state data of token1.
[0154] For a given node token, the current GPU sends the hidden state data of the token to the local GPU via intra-node communication. The local GPU then uses the corresponding local expert to perform expert computation on the data. For example, GPU1 sends the hidden state data of token2 to GPU2, and GPU2 uses EP2 to perform expert computation on the hidden state data of token2.
[0155] Specifically, intra-node communication can be NVLink communication. NVLink is a high-speed communication method between multiple GPUs within a node.
[0156] In this embodiment, the current GPU can perform expert calculations on the local token and distribute the local node token to the local node GPU, thereby improving processing efficiency and accuracy.
[0157] Figure 5 Based on a schematic diagram of the fifth embodiment of this disclosure, this embodiment provides a data processing method. This method is applied to current GPUs, such as... Figure 5 As shown, the method includes:
[0158] 501. Receive a token sequence, the token sequence including the hidden state data of multiple tokens.
[0159] 502. Determine the target expert index corresponding to the token sequence.
[0160] 503. Send the target expert index and data to the CPU so that the CPU performs a communication calculation process based on the target expert index and sends the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata.
[0161] The data includes: the hidden state data of all tokens in the token sequence; the target expert of the cross-node token includes a cross-node expert, which is deployed on the cross-node GPU.
[0162] This example is executed by the current GPU.
[0163] After receiving the token sequence, the GPU can use a gating network to calculate the target expert corresponding to each token, and then form a target expert index by mapping all tokens and their target experts.
[0164] Then, the GPU sends the target expert index and the hidden state data of all tokens in the token sequence to the CPU so that the CPU can perform communication calculations and related processing.
[0165] In this embodiment, the target expert index and data are sent to the CPU by the GPU, triggering the CPU to perform communication calculations. This can offload communication calculation tasks and cross-node communication tasks from the GPU to the CPU, allowing the GPU to focus on expert calculation tasks or intra-node communication tasks, thereby balancing the resource utilization of each hardware and improving the overall throughput.
[0166] The above example uses the CPU for communication-related processing. In actual implementation, the process can be switched based on the GPU's load. When the load is high, the CPU handles the communication-related processing; when the load is low, the GPU handles it.
[0167] For example, the GPU can obtain the number of tokens currently received. When the number is greater than a preset threshold, it sends the target expert index and data to the CPU, which then performs communication calculations and related processing. When the number is less than or equal to the preset threshold, the GPU performs the communication calculations and related processing itself.
[0168] This allows the GPU or CPU to handle communication-related processing based on actual needs, improving flexibility.
[0169] In some embodiments, the method may further include:
[0170] A local token is determined from the token sequence, and the target expert corresponding to the local token includes a local expert, which is deployed on the current GPU.
[0171] The local expert is used to perform expert calculations on the local token.
[0172] After the CPU calculates the communication metadata, it can send the communication metadata to the current GPU. The current GPU can determine the local token from the token sequence based on the communication metadata. The target expert of the local token includes the local expert, and the local expert is deployed on the current GPU.
[0173] For example, if the current GPU is GPU1, the token sequence includes token1, the target expert corresponding to token1 includes EP1, and EP1 is deployed on GPU1, then token1 is a local token.
[0174] Then, GPU1 retrieves the hidden state data of token1 from the video memory and uses EP1 to perform expert calculations on the hidden state data of token1.
[0175] In some embodiments, the method may further include:
[0176] The local node token is determined from the token sequence, and the target expert corresponding to the local node token includes the local node expert, which is deployed on the local node GPU.
[0177] The hidden state data of the local node token is sent to the local node GPU via the intra-node communication bus, so that the local node GPU can use the local node expert to perform expert calculations on the local node token.
[0178] After the CPU calculates the communication metadata, it can send the communication metadata to the current GPU. The current GPU can determine the local node token from the token sequence based on the communication metadata. The target experts of the local node token include the local node experts, and the local node experts are deployed on the local node GPU.
[0179] For example, if the current GPU is GPU1, the token sequence includes token2, the target expert corresponding to token2 includes EP2, EP2 is deployed on GPU2, and GPU2 and GPU1 are located in the same node, then token2 is the token of this node.
[0180] Next, GPU1 retrieves the hidden state data of token2 from the video memory and sends it to GPU2 through intra-node communication. GPU2 then uses EP2 to perform expert calculations on the hidden state data of token2.
[0181] The intra-node communication mentioned above can specifically be NVlink communication.
[0182] In this embodiment, the current GPU can perform expert calculations on the local token and distribute the local node token to the local node GPU, thereby improving processing efficiency and accuracy.
[0183] Figure 6 This is a schematic diagram based on the sixth embodiment of the present disclosure, which provides a communication device. This device is applied to a CPU, such as... Figure 6 As shown, the device 600 includes: a receiving module 601, a storage module 602, a computing module 603, and a communication module 604.
[0184] The receiving module 601 is used to receive the target expert index and data sent by the current GPU; the target expert index is determined by the current GPU based on the token sequence, and the data includes: the hidden state data of all tokens in the token sequence;
[0185] Storage module 602 is used to store the data in the memory of the CPU;
[0186] The computing module 603 is used to perform a communication computing process based on the target expert index to obtain communication metadata; the communication metadata includes: target storage information and cross-node GPU information, the target storage information is the storage information of the hidden state data of the cross-node token in the memory, the target expert of the cross-node token includes the cross-node expert, and the cross-node expert is deployed on the cross-node GPU;
[0187] The communication module 604 is used to send the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata.
[0188] In this embodiment, the CPU receives the target expert index and data sent by the current GPU, performs the communication calculation process based on the target expert index, and performs cross-node communication based on the communication metadata. This can offload the communication calculation task and the cross-node communication task from the GPU to the CPU, allowing the GPU to focus on expert calculation tasks or intra-node communication tasks, thereby balancing the resource utilization of each hardware and improving the overall throughput.
[0189] In some embodiments, the communication module 604 is further configured to:
[0190] Create a task unit, which contains the target storage information and the cross-node GPU information;
[0191] In response to the task unit meeting preset conditions, a trigger command is sent to the network card, so that the network card obtains the task unit based on the trigger command, and sends the hidden state data of the cross-node token to the cross-node GPU according to the task unit.
[0192] In this embodiment, by sending a trigger command to the network card when the task unit meets the preset conditions, the network card can be triggered to perform batch communication processing, thereby reducing resource overhead and improving communication efficiency.
[0193] In some embodiments, the communication module 604 is further configured to:
[0194] The target expert index and the data sent by the current GPU are received via the first PCIe bus.
[0195] In some embodiments, the communication module 604 is further configured to:
[0196] A trigger command is sent to the network card via the second PCIe bus.
[0197] In this embodiment, the GPU, CPU, and network card communicate via the PCIe bus, which enables simple and efficient communication between them.
[0198] In some embodiments, the network interface card (NIC) is an RNIC; the hidden state data of the cross-node token is sent by the RNIC to the cross-node GPU based on RDMA communication.
[0199] In this embodiment, cross-node communication based on RDAM technology can further reduce communication latency and improve inference or training speed.
[0200] In some embodiments, the CPU includes multiple thread pools; the computing module 603 is further configured to:
[0201] The target expert index is divided into multiple groups;
[0202] Each thread pool is used to index each group of target experts and execute the communication calculation process in parallel.
[0203] In this embodiment, processing efficiency can be improved based on the above-described parallel execution process.
[0204] Figure 7 This is a schematic diagram based on the seventh embodiment of the present disclosure, which provides a data processing apparatus. This apparatus is applied to current GPUs, such as... Figure 7 As shown, the device 700 includes: a receiving module 701, a determining module 702, and a sending module 703.
[0205] The receiving module 701 is used to receive a token sequence, the token sequence including the hidden state data of multiple tokens;
[0206] The determining module 702 is used to determine the target expert index corresponding to the token sequence;
[0207] The sending module 703 is used to send the target expert index and data to the CPU, so that the CPU performs a communication calculation process based on the target expert index and sends the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata.
[0208] The data includes: the hidden state data of all tokens in the token sequence;
[0209] The target experts for the cross-node token include cross-node experts, which are deployed on the cross-node GPU.
[0210] In this embodiment, the target expert index and data are sent to the CPU by the GPU, triggering the CPU to perform communication calculations. This can offload communication calculation tasks and cross-node communication tasks from the GPU to the CPU, allowing the GPU to focus on expert calculation tasks or intra-node communication tasks, thereby balancing the resource utilization of each hardware and improving the overall throughput.
[0211] In some embodiments, the device 700 further includes:
[0212] A local computing module is used to determine a local token in the token sequence, wherein the target expert corresponding to the local token includes a local expert, and the local expert is deployed on the current GPU; and to perform expert computing on the local token using the local expert.
[0213] In this embodiment, the current GPU can perform expert calculations on the local token, improving processing efficiency and accuracy.
[0214] In some embodiments, the device 700 further includes:
[0215] An intra-node communication module is used to determine the local node token in the token sequence, wherein the target expert corresponding to the local node token includes the local node expert, and the local node expert is deployed on the local node GPU; and to send the hidden state data of the local node token to the local node GPU through the intra-node communication bus, so that the local node GPU uses the local node expert to perform expert calculation on the local node token.
[0216] In this embodiment, the current GPU can distribute the local node token to other local GPUs, thereby improving processing efficiency and accuracy.
[0217] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.
[0218] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.
[0219] It is understandable that, unless otherwise specified, the order of steps in the process indicates that the temporal relationship between these steps is not limited.
[0220] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0221] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0222] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0223] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0224] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0225] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as communication methods or data processing methods. For example, in some embodiments, the communication method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the communication method or data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the communication method or data processing method by any other suitable means (e.g., by means of firmware).
[0226] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0227] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable task processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0228] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0230] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0231] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0232] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0233] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A communication method applied to a CPU, the method comprising: Receive the target expert index and data sent by the current GPU; The target expert index is determined by the current GPU based on the token sequence, and the data includes: the hidden state data of all tokens in the token sequence; The data is stored in the CPU's memory; Based on the target expert index, a communication computation process is performed to obtain communication metadata; the communication metadata includes: target storage information and cross-node GPU information, the target storage information is the storage information of the hidden state data of the cross-node token in the memory, the target expert of the cross-node token includes the cross-node expert, and the cross-node expert is deployed on the cross-node GPU; Based on the communication metadata, the hidden state data of the cross-node token stored in memory is sent to the cross-node GPU.
2. The method according to claim 1, wherein, The step of sending the hidden state data of the cross-node token stored in memory to the cross-node GPU based on the communication metadata includes: Create a task unit, which contains the target storage information and the cross-node GPU information; In response to the task unit meeting preset conditions, a trigger command is sent to the network card, so that the network card obtains the task unit based on the trigger command, retrieves the hidden state data of the cross-node token from the memory according to the task unit, and sends it to the cross-node GPU.
3. The method according to claim 2, wherein, The target expert index and the data are sent by the current GPU to the CPU via the first PCIe bus; The trigger command is sent by the CPU to the network card through the second PCIe bus; The hidden state data of the cross-node token is obtained by the network card from the memory of the CPU through the second PCIe bus.
4. The method according to claim 2, wherein, The network interface card is an RNIC; The hidden state data of the cross-node token is sent by the RNIC to the cross-node GPU based on RDMA communication.
5. The method according to any one of claims 1-4, wherein, The CPU includes multiple thread pools; The communication calculation process based on the target expert index includes: The target expert index is divided into multiple groups; Each thread pool is used to index each group of target experts and execute the communication calculation process in parallel.
6. A data processing method applied to a current GPU, the method comprising: Receive a token sequence, the token sequence including the hidden state data of multiple tokens; Determine the target expert index corresponding to the token sequence; The target expert index and data are sent to the CPU, so that the CPU performs a communication calculation process based on the target expert index and sends the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata. The data includes: the hidden state data of all tokens in the token sequence; The target experts for the cross-node token include cross-node experts, which are deployed on the cross-node GPU.
7. The method according to claim 6, further comprising: A local token is determined from the token sequence, and the target expert corresponding to the local token includes a local expert, which is deployed on the current GPU. The local expert is used to perform expert calculations on the local token.
8. The method according to claim 6, further comprising: The local node token is determined from the token sequence, and the target expert corresponding to the local node token includes the local node expert, which is deployed on the local node GPU. The hidden state data of the local node token is sent to the local node GPU via the intra-node communication bus, so that the local node GPU can use the local node expert to perform expert calculations on the local node token.
9. A communication device applied to a CPU, the device comprising: The receiving module is used to receive the target expert index and data sent by the current GPU; The target expert index is determined by the current GPU based on the token sequence, and the data includes: the hidden state data of all tokens in the token sequence; A storage module is used to store the data in the memory of the CPU; The computing module is used to perform a communication computing process based on the target expert index to obtain communication metadata; the communication metadata includes: target storage information and cross-node GPU information, the target storage information is the storage information of the hidden state data of the cross-node token in the memory, the target expert of the cross-node token includes the cross-node expert, and the cross-node expert is deployed on the cross-node GPU; A communication module is used to send the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata.
10. A data processing apparatus applied to a current GPU, the apparatus comprising: A receiving module is used to receive a token sequence, the token sequence including the hidden state data of multiple tokens; The determination module is used to determine the target expert index corresponding to the token sequence; The sending module is used to send the target expert index and data to the CPU, so that the CPU can perform a communication calculation process based on the target expert index, and send the hidden state data of the cross-node token stored in memory to the cross-node GPU according to the communication metadata. The data includes: the hidden state data of all tokens in the token sequence; The target experts for the cross-node token include cross-node experts, which are deployed on the cross-node GPU.
11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.
13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.