A tensor aggregation method, device, storage medium and program product
By executing intra-node and inter-node aggregation tasks in parallel, and utilizing the central processing unit and remote direct memory access technology, the problems of communication latency and low bandwidth utilization in the MoE architecture are solved, thereby improving overall performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHUHAI BIREN INTEGRATED CIRCUIT CO LTD
- Filing Date
- 2026-05-15
- Publication Date
- 2026-07-14
AI Technical Summary
Under the MoE architecture, serial aggregation results in significant communication latency and low network bandwidth utilization, impacting overall performance.
The tensor aggregation process is pipelined in parallel by employing parallel execution of intra-node and inter-node aggregation tasks and by using central processing unit scheduling and remote direct memory access technology.
It effectively reduced communication and aggregation time, improved network bandwidth utilization, and enhanced overall performance.
Smart Images
Figure CN122387907A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a tensor aggregation method, device, storage medium, and program product. Background Technology
[0002] In the fields of artificial intelligence computing and distributed systems technology, the Mixture of Experts (MoE) architecture is widely used in inference scenarios to improve model capacity and inference efficiency. In a MoE architecture, each node includes multiple AI chips (e.g., Graphics Processing Units (GPUs)), and each GPU deploys one or more expert networks. Tokens are assigned to expert networks on different nodes for processing, and the processing results from each expert network are then aggregated to obtain the final aggregated result.
[0003] However, in actual aggregation, the serial aggregation method is used to collect processing results from each expert network in turn, and then the collected processing results are aggregated. This serial aggregation method has a large communication delay, resulting in low network bandwidth utilization and thus affecting the overall performance. Summary of the Invention
[0004] This invention provides a tensor aggregation method, device, storage medium, and program product to reduce communication latency in tensor aggregation under the MoE architecture, improve network bandwidth utilization, and thus affect overall performance.
[0005] On one hand, embodiments of this application provide a tensor aggregation method applied to a first node, the method comprising: Execute intra-node aggregation tasks and inter-node aggregation tasks in parallel; The node-in-node aggregation task includes the following operations: obtaining the first intra-card tensors generated by multiple expert networks that process the first word; aggregating the obtained multiple first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to the second node, wherein the multiple expert networks are deployed in multiple artificial intelligence chips in the first node; The inter-node aggregation task includes the following operations: receiving a second inter-card tensor generated for the second word from the second node; obtaining a third inter-card tensor generated by the first node for the second word; and aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word.
[0006] On one hand, embodiments of this application provide a tensor aggregation apparatus applied to a first node. This apparatus is used to execute intra-node aggregation tasks and inter-node aggregation tasks in parallel, including: The first aggregation module is used to perform the following operations in the node aggregation task: obtaining the first intra-card tensor generated by multiple expert networks that process the first word; aggregating the obtained multiple first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to the second node, wherein the multiple expert networks are deployed in multiple artificial intelligence chips in the first node; The second aggregation module is used to perform the following operations for the inter-node aggregation task: receiving a second inter-card tensor generated for the second word sent by the second node; obtaining a third inter-card tensor generated by the first node for the second word; and aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word.
[0007] Optionally, the first aggregation module is specifically used for: Perform the following operations via the main thread: Obtain the original in-card tensors output by the multiple expert networks that process the first word; The original layout type of each original card tensor is rearranged into the target layout type to obtain the corresponding first card tensor. The original layout type is a layout type supported by the artificial intelligence chip, and the target layout type is a layout type supported by the central processing unit.
[0008] Optionally, the second aggregation module is specifically used for: The second inter-card tensor and the third inter-card tensor are aggregated to obtain a preliminary aggregation result, which corresponds to the target layout type. The target layout type corresponding to the preliminary aggregation result is rearranged into the original layout type by the main thread to obtain the target aggregation result.
[0009] Optionally, the first aggregation module is specifically used for: The first thread aggregates the multiple intra-card tensors to obtain the inter-card tensor. The first inter-card tensor is pushed into the first queue to trigger the second thread to send the first inter-card tensor in the first queue to the second node via remote direct memory access.
[0010] Optionally, the second aggregation module is specifically used for: The third thread uses remote direct memory access to receive the second inter-card tensor sent by the second node.
[0011] Optionally, the second aggregation module is specifically used for: The third thread pushes the second inter-card tensor into the second queue to trigger the fourth thread to obtain the second intra-card tensors generated by multiple expert networks that process the second terminology, and aggregates the obtained multiple second intra-card tensors to obtain the third inter-card tensor.
[0012] Optionally, the second aggregation module is specifically used for: The second inter-card tensor is saved to shared memory via the third thread; The fourth thread reads the second inter-card tensor from the shared memory and aggregates the second inter-card tensor with the third inter-card tensor to obtain the target aggregation result.
[0013] On one hand, embodiments of this application provide a computer device, including: Memory, used to store program instructions; An artificial intelligence chip is used to run program instructions stored in the memory and execute the steps of the tensor aggregation method described above according to the obtained program.
[0014] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer to perform the steps of the tensor aggregation method described above.
[0015] On one hand, embodiments of this application provide a computer program product, including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the tensor aggregation method described above.
[0016] In this embodiment, intra-node aggregation tasks and inter-node aggregation tasks are executed in parallel. The intra-node aggregation task includes the following operations: acquiring first intra-card tensors generated by multiple expert networks processing the first term; aggregating the acquired first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to a second node. Multiple expert networks are deployed in multiple AI chips within the first node. The inter-node aggregation task includes the following operations: receiving a second inter-card tensor generated by the second node for the second term; acquiring a third inter-card tensor generated by the first node for the second term; aggregating the second and third inter-card tensors to obtain the target aggregation result corresponding to the second term. This achieves pipelined parallelism between intra-node and inter-node aggregation, resulting in overlapping execution times between the two tasks. This effectively reduces communication and aggregation time, thereby reducing overall communication latency, improving network bandwidth utilization, and ultimately enhancing overall performance. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1A This is a schematic diagram of a chip architecture provided in an embodiment of this application; Figure 1B This application provides a schematic diagram of the structure of a node according to an embodiment of the present application; Figure 2 This is a schematic diagram of a tensor aggregation method provided in an embodiment of this application; Figure 3 A flowchart illustrating another tensor aggregation method provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a tensor polymerization device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.
[0020] refer to Figure 1A This is a schematic diagram of the structure of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be streaming processing clusters (SPCs). The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.
[0021] In this embodiment of the application, one or more expert networks under the MoE architecture can be deployed in the artificial intelligence chip 100.
[0022] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.
[0023] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-purpose computing on graphics processing units (GPGPU), Domain Specific Architecture (DSA), etc.
[0024] refer to Figure 1B This is a schematic diagram of the structure of a node applicable to an embodiment of this application. In actual applications, node 103 can be a server, terminal device, etc.; the node 103 includes a central processing unit 104 and multiple artificial intelligence chips 100.
[0025] In node 103, the central processing unit 104 can communicate with any one of the artificial intelligence chips 100. Of course, any two artificial intelligence chips 100 can also communicate with each other. This application does not make any specific limitations on this.
[0026] In this embodiment, the MoE architecture can be deployed on multiple nodes 103. The first node can be any one of the multiple nodes 103, and the second node can be any one of the multiple nodes 103 other than the first node. It should be noted that the multiple nodes 103 may include other nodes besides the first and second nodes, but this application does not specifically limit this.
[0027] The following is based on Figure 1A The diagram shows the architecture of the artificial intelligence chip. Figure 1B The diagram shown illustrates the node architecture and provides a detailed description of a tensor aggregation method offered in this application. (See also...) Figure 2 The process of this method is executed by the first node, and the method includes the following steps: Step 201: Execute intra-node aggregation tasks and inter-node aggregation tasks in parallel.
[0028] Specifically, the tensor aggregation method of this application can be applied to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the tensors used for aggregation can differ depending on the application scenario.
[0029] For example, in a text processing scenario, the aggregated tensor can be text data used in tasks such as text generation and text recognition.
[0030] For example, in a speech processing scenario, the aggregated tensor can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.
[0031] For example, in image processing scenarios, the aggregated tensors can be image data used in tasks such as image preprocessing, image segmentation, and object detection.
[0032] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.
[0033] In this embodiment, the MoE architecture is deployed across multiple nodes, each containing multiple expert networks within the MoE architecture. These expert networks can be deployed on multiple AI chips within the node. For any given word, a router distributes the word to multiple expert networks within the MoE architecture for parallel processing. The processing results obtained by each expert network are then aggregated to obtain the aggregated result corresponding to that word. This aggregated result can be output to the next layer network for further processing or output directly.
[0034] In this embodiment, lexical units and tensors represent the same semantic unit at different stages of the model. The user's input text is segmented to obtain lexical units, which are then input into the model and mapped to corresponding embedding tensors through the model's embedding layer. Next, the embedding tensor is further processed by other network layers within the model. During this processing, the input and output of each network layer are both tensors. For each expert network in this application, the input can be an embedding tensor or the output tensor of the previous network layer. The processing result of the expert network (i.e., the output of the expert network) is also a tensor, and the aggregated result obtained by aggregating the processing results of multiple expert networks is also a tensor.
[0035] In practical applications, multiple expert networks that process lexical units may be distributed across multiple nodes. Thus, the processing results obtained by each expert network within each node are first aggregated, and this aggregation process is called intra-node aggregation task. Then, the aggregation results obtained by each of the multiple nodes are aggregated again, and this aggregation process is called inter-node aggregation task.
[0036] Step 202, the intra-node aggregation task includes the following operations: obtaining the first intra-card tensors generated by multiple expert networks that process the first word; aggregating the obtained multiple first intra-card tensors to obtain the first inter-card tensor, and sending the first inter-card tensor to the second node.
[0037] In this embodiment, the expert network is the sub-network that actually processes the data in the MoE architecture. Multiple expert networks processing the first term are deployed on multiple AI chips in the first node, wherein one AI chip can deploy one or more expert networks. The number of expert networks deployed on each AI chip in the first node can be different or the same. The multiple expert networks processing the first term can be all or some of the expert networks deployed in the first node. The network structures of the expert networks deployed in the first node are the same, but their network parameters are different.
[0038] For example, the first node includes 4 AI chips and deploys 8 expert networks, with 2 expert networks deployed in each AI chip.
[0039] For example, the first node includes four AI chips and deploys four expert networks, with one expert network deployed in each AI chip. This application does not impose specific limitations on this.
[0040] In some embodiments, the tensor aggregation method in this application can be executed by the central processing unit (CPU) in the first node, wherein the CPU includes a main thread. The main thread is responsible for scheduling aggregation tasks within the node and aggregation tasks between nodes. Furthermore, since the AI chip and the CPU support different data layout types, the CPU rearranges the layout types of the tensors passed by the AI chip before executing tensor aggregation, as follows: The main thread performs the following operations: obtain the original in-card tensors output by multiple expert networks that process the first word; rearrange the original layout type of each original in-card tensor into the target layout type to obtain the corresponding first in-card tensor. The original layout type is the layout type supported by the artificial intelligence chip, and the target layout type is the layout type supported by the central processing unit.
[0041] Specifically, the first word in the input is mapped through the model's embedding layer to obtain the embedding tensor corresponding to the first word. In practical applications, the embedding tensor can be input into an expert network for processing to obtain the original in-card tensor. Alternatively, the embedding tensor can be processed by at least one other network layer in the model to obtain the output tensor, and then the output tensor can be input into an expert network for processing to obtain the original in-card tensor.
[0042] Each AI chip transmits the obtained raw tensors within the card to the central processing unit (CPU). The CPU rearranges the layout of the raw tensors within the card to obtain the first tensor within the card.
[0043] In some embodiments, the central processing unit further includes a first thread and a second thread. The first thread aggregates the acquired multiple intra-card tensors to obtain an inter-card tensor; then, the inter-card tensor is pushed into a first queue to trigger the second thread to send the inter-card tensor in the first queue to the second node via remote direct memory access.
[0044] Specifically, Remote Direct Memory Access (RDMA) is a technology in which the central processing unit of one node bypasses the central processing unit of another node and directly reads and writes data to the artificial intelligence chip corresponding to that other node.
[0045] In the node-to-node aggregation task of this application, the second thread directly writes the inter-card tensor of the first node into the artificial intelligence chip that processes the first word in the second node via RDMA.
[0046] In this embodiment, the first node achieves efficient communication with the second node based on RDMA technology, reducing the communication time between the first node and the second node, thereby improving the performance of the first node in processing tensor aggregation.
[0047] In addition, before the main thread schedules the first thread to aggregate multiple tensors obtained from the first card, the main thread uses barrier instructions supported by each AI chip in the first node to synchronize the outputs of multiple expert networks within the first node; that is, before aggregating multiple tensors from the first card, the barrier instructions ensure that each expert network has processed and obtained the corresponding tensors from the first card.
[0048] In this embodiment, the barrier instruction ensures that all tensors in each card that need to be aggregated have been obtained, thereby guaranteeing the correctness of the aggregation task within the node.
[0049] Step 203, the inter-node aggregation task includes the following operations: receiving the second inter-card tensor generated for the second word sent by the second node; obtaining the third inter-card tensor generated by the first node for the second word; aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word.
[0050] In specific implementation, the first node can receive the second inter-card tensor generated for the second word element from one or more second nodes; at the same time, it can obtain the third inter-card tensor generated by aggregation within the local execution node for the second word element; and then aggregate the obtained second inter-card tensors and the third inter-card tensor to obtain the target aggregation result corresponding to the two words element.
[0051] In this embodiment, intra-node aggregation tasks and inter-node aggregation tasks are executed in parallel. The intra-node aggregation task includes the following operations: acquiring first intra-card tensors generated by multiple expert networks processing the first term; aggregating the acquired first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to a second node. Multiple expert networks are deployed in multiple AI chips within the first node. The inter-node aggregation task includes the following operations: receiving a second inter-card tensor generated by the second node for the second term; acquiring a third inter-card tensor generated by the first node for the second term; aggregating the second and third inter-card tensors to obtain the target aggregation result corresponding to the second term. This achieves pipelined parallelism between intra-node and inter-node aggregation, resulting in overlapping execution times between the two tasks. This effectively reduces communication and aggregation time, thereby reducing overall communication latency, improving network bandwidth utilization, and ultimately enhancing overall performance.
[0052] In some embodiments, the second inter-card tensor sent by the second node is received by a third thread using remote direct memory access.
[0053] Specifically, the first queue is a low-latency first-in-first-out queue (or simply low-latency FIFO). The characteristic of low-latency FIFO is that once the low-latency FIFO is not empty, subsequent operations are triggered as quickly as possible. In this embodiment, when the first inter-card tensor is pushed into the first queue, the second thread is immediately triggered to communicate the first inter-card tensor from the first node to the second node.
[0054] In the inter-node aggregation task of this application, the third thread receives the second inter-card tensor sent by the central processing unit of the second node via RDMA. The second inter-card tensor is obtained by the central processing unit of the second node executing the intra-node aggregation task.
[0055] In this embodiment, by pushing the first inter-card tensor into a low-latency FIFO, the scheduling time between the first thread and the second thread in the intra-node aggregation task is reduced, and asynchronous communication between different threads is realized. At the same time, the third thread realizes efficient communication with the second node based on RDMA technology in the inter-node aggregation task, reducing the communication time between the first node and the second node, thereby improving the performance of the first node in processing tensor aggregation.
[0056] In some embodiments, the second inter-card tensor is pushed into the second queue by the third thread to trigger the fourth thread to obtain the second intra-card tensors generated by multiple expert networks that process the second terminology, and to aggregate the obtained multiple second intra-card tensors to obtain the third inter-card tensor.
[0057] Specifically, the second queue is a low-latency, first-in-first-out queue, with the same functional characteristics as the first queue, and will not be elaborated further here. Multiple expert networks processing the second word are deployed across multiple AI chips in the second node, where each AI chip can deploy one or more expert networks. These multiple expert networks processing the second word can be all or some of the expert networks deployed in the second node. It should be noted that the multiple expert networks processing the second word can be the same as or different from the multiple expert networks processing the first word.
[0058] It should be noted that after the second inter-card tensor is pushed into the second queue by the third thread, the fourth thread can also be triggered to directly obtain the third inter-card tensor. In other words, the third inter-card tensor can also be obtained by the fourth thread during other idle time by obtaining the second intra-card tensors generated by multiple expert networks processing the second terminology, and aggregating the multiple second intra-card tensors.
[0059] In the embodiments of this application, the first word element and the second word element can be the same word element or different word elements.
[0060] When the first word element and the second word element are the same word element, the first node executes the intra-node aggregation task and the inter-node aggregation task for the first word element in parallel.
[0061] Specifically, when the first and second tokens are the same token, the first intra-card tensor is the second intra-card tensor, and the first inter-card tensor is the third inter-card tensor. That is, while the first thread aggregates multiple first intra-card tensors to obtain the first inter-card tensor (i.e., the third inter-card tensor), the third thread simultaneously uses remote direct memory access to receive the second inter-card tensor sent by the second node; thus, the aggregation process of obtaining the first inter-card tensor overlaps with the communication process of receiving the second inter-card tensor. Then, the second and third inter-card tensors (i.e., the first inter-card tensor) are aggregated to obtain the target aggregation result corresponding to the second token (i.e., the first token).
[0062] When the first word and the second word are different words, the first node performs the intra-node aggregation task for the first word and the inter-node aggregation task for the second word in parallel.
[0063] In other words, the CPU of the first node executes the intra-node aggregation task for the first word through the main thread, the first thread, and the second thread; simultaneously, the CPU of the first node executes the inter-node aggregation task for the second word through the main thread, the third thread, and the fourth thread. The parallel execution of the intra-node aggregation task and the inter-node aggregation task overlaps in execution time but does not interfere with each other.
[0064] In some embodiments, the second inter-card tensor is saved to shared memory by a third thread; the second inter-card tensor is read from shared memory by a fourth thread, and the second inter-card tensor is aggregated with the third inter-card tensor to obtain the target aggregation result.
[0065] Specifically, the threads within the central processing unit synchronize through shared memory. In the inter-node aggregation task, the third thread saves the second inter-card tensor obtained from the second node to the second queue in shared memory, so that the fourth thread can directly obtain the second inter-card tensor from shared memory, and aggregate the second inter-card tensor with the third inter-card tensor to obtain the target aggregation result.
[0066] Alternatively, if the fourth thread has already aggregated multiple intra-card tensors during other idle time to obtain the third inter-card tensor before the third thread receives the second inter-card tensor, then the third inter-card tensor can also be saved to shared memory first. In this way, after the third thread receives the second inter-card tensor sent by the second node and saves it to shared memory, the fourth thread can directly read the second inter-card tensor and the third inter-card tensor from shared memory to aggregate them and obtain the target aggregation result.
[0067] In this embodiment, multiple threads of the central processing unit can synchronize and exchange data through shared memory, which effectively improves the synchronization efficiency in a multi-threaded environment, while also improving data exchange efficiency and pipeline throughput.
[0068] In some embodiments, the inter-card tensor and the inter-card tensor are aggregated to obtain a preliminary aggregation result; then, the target layout type corresponding to the preliminary aggregation result is rearranged to the original layout type through the main thread to obtain the target aggregation result.
[0069] Specifically, after the CPU aggregates the inter-card tensors of the second and third cards, the preliminary aggregation result corresponds to the target layout type, i.e., the layout type supported by the CPU. Since the preliminary aggregation result obtained by the CPU may need to be returned to the AI chip for further inference, the CPU rearranges the layout of the preliminary aggregation result through the main thread to obtain the target aggregation result of the original layout type, i.e., the target aggregation result of the layout type supported by the AI chip, to ensure the accuracy of subsequent processing by the AI chip.
[0070] It should be noted that the second node can also execute intra-node aggregation tasks and inter-node aggregation tasks in parallel. The inter-node aggregation task specifically includes the following steps: the central processing unit receives the first inter-card tensor generated for the first word sent by the first node; obtains the fourth inter-card tensor generated by the second node for the first word; aggregates the first inter-card tensor and the fourth inter-card tensor to obtain the target aggregation result corresponding to the first word.
[0071] The intra-node aggregation task specifically includes the following steps: obtaining the intra-card tensors generated by multiple expert networks that process the third term in the second node, aggregating the obtained intra-card tensors to obtain the inter-card tensor, and sending the inter-card tensor to the first node.
[0072] The specific execution process is the same as the process described above where the first node executes the intra-node aggregation task and the inter-node aggregation task in parallel, so it will not be repeated here.
[0073] To more intuitively illustrate the parallel processes of intra-node and inter-node aggregation tasks, the following section uses the central processing unit of the first node as an example to introduce the multi-threaded pipeline design. See [link to documentation] for details. Figure 3 : The central processing unit includes: the main thread, the first thread, the second thread, the third thread, and the fourth thread. These threads work together to complete the aggregation tasks within the node and the aggregation tasks between the nodes.
[0074] The node-level aggregation task includes: the main thread reads the original card tensor corresponding to the first term from the device to the host (i.e., from each AI chip processing the first term in the first node to the central processing unit), and then rearranges the layout of the original card tensor corresponding to the first term, that is, rearranges the original card tensor of the original layout type into the first card tensor of the target layout type. After synchronization via barrier instructions, the first thread is called to aggregate multiple first card tensors into a first inter-card tensor, and then the second thread is triggered to send the first inter-card tensor to the second node via RDMA.
[0075] The inter-node aggregation task includes: the third thread receiving the second inter-card tensor generated by the second node; triggering the fourth thread to prepare the third inter-card data, aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word; the main thread rearranging the layout of the target aggregation result corresponding to the second word, rearranging the target aggregation result of the target layout type to the original layout type, and saving the target aggregation result of the original layout type from the host to the device (i.e., from the central processing unit to the artificial intelligence chip).
[0076] according to Figure 3 As can be seen, the intra-node aggregation task and inter-node aggregation in this application are executed in parallel. By executing intra-node aggregation tasks and inter-node aggregation tasks in parallel by different threads, the overall aggregation process under the MoE architecture is decomposed into multiple pipeline tasks, so that different pipeline tasks overlap in part of the time, thereby improving network bandwidth utilization and overall performance.
[0077] Based on the same technical concept, this application provides a structural schematic diagram of a tensor polymerization device 400, as shown below. Figure 4As shown, the device is applied to the first node, specifically for parallel execution of intra-node aggregation tasks and inter-node aggregation tasks, including: The first aggregation module 401 is used to perform the following operations in the node aggregation task: obtaining the first intra-card tensor generated by multiple expert networks that process the first word; aggregating the obtained multiple first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to the second node, wherein the multiple expert networks are deployed in multiple artificial intelligence chips in the first node; The second aggregation module 402 is used to perform the following operations in the inter-node aggregation task: receiving a second inter-card tensor generated for the second word sent by the second node; obtaining a third inter-card tensor generated by the first node for the second word; and aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word.
[0078] Optionally, the first aggregation module 401 is specifically used for: Perform the following operations via the main thread: Obtain the original in-card tensors output by the multiple expert networks that process the first word; The original layout type of each original card tensor is rearranged into the target layout type to obtain the corresponding first card tensor. The original layout type is a layout type supported by the artificial intelligence chip, and the target layout type is a layout type supported by the central processing unit.
[0079] Optionally, the second aggregation module 402 is specifically used for: The second inter-card tensor and the third inter-card tensor are aggregated to obtain a preliminary aggregation result, which corresponds to the target layout type. The target layout type corresponding to the preliminary aggregation result is rearranged into the original layout type by the main thread to obtain the target aggregation result.
[0080] Optionally, the first aggregation module 401 is specifically used for: The first thread aggregates the multiple intra-card tensors to obtain the inter-card tensor. The first inter-card tensor is pushed into the first queue to trigger the second thread to send the first inter-card tensor in the first queue to the second node via remote direct memory access.
[0081] Optionally, the second aggregation module 402 is specifically used for: The third thread uses remote direct memory access to receive the second inter-card tensor sent by the second node.
[0082] Optionally, the second aggregation module 402 is specifically used for: The third thread pushes the second inter-card tensor into the second queue to trigger the fourth thread to obtain the second intra-card tensors generated by multiple expert networks that process the second terminology, and aggregates the obtained multiple second intra-card tensors to obtain the third inter-card tensor.
[0083] Optionally, the second aggregation module 402 is specifically used for: The second inter-card tensor is saved to shared memory via the third thread; The fourth thread reads the second inter-card tensor from the shared memory and aggregates the second inter-card tensor with the third inter-card tensor to obtain the target aggregation result.
[0084] In this embodiment, intra-node aggregation tasks and inter-node aggregation tasks are executed in parallel. The intra-node aggregation task includes the following operations: acquiring first intra-card tensors generated by multiple expert networks processing the first term; aggregating the acquired first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to a second node. Multiple expert networks are deployed in multiple AI chips within the first node. The inter-node aggregation task includes the following operations: receiving a second inter-card tensor generated by the second node for the second term; acquiring a third inter-card tensor generated by the first node for the second term; aggregating the second and third inter-card tensors to obtain the target aggregation result corresponding to the second term. This achieves pipelined parallelism between intra-node and inter-node aggregation, resulting in overlapping execution times between the two tasks. This effectively reduces communication and aggregation time, thereby reducing overall communication latency, improving network bandwidth utilization, and ultimately enhancing overall performance.
[0085] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited. Figure 5 Taking the connection between the AI chip 100 and the memory 501 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.
[0086] In this embodiment of the application, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 501, the at least one artificial intelligence chip 100 can perform the steps of the tensor aggregation method described above.
[0087] The artificial intelligence chip 100 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It performs tensor aggregation by running or executing instructions stored in the memory 501 and accessing data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units. It may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.
[0088] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0089] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 501 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0090] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the tensor aggregation method described above.
[0091] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the tensor aggregation method described above.
[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0096] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations. Those skilled in the art should understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0097] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A tensor aggregation method, characterized in that, Applied to the first node, including: Execute intra-node aggregation tasks and inter-node aggregation tasks in parallel; The node-in-node aggregation task includes the following operations: obtaining the first intra-card tensors generated by multiple expert networks that process the first word; aggregating the obtained multiple first intra-card tensors to obtain a first inter-card tensor, and sending the first inter-card tensor to the second node, wherein the multiple expert networks are deployed in multiple artificial intelligence chips in the first node; The inter-node aggregation task includes the following operations: receiving a second inter-card tensor generated for the second word from the second node; obtaining a third inter-card tensor generated by the first node for the second word; and aggregating the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word.
2. The method as described in claim 1, characterized in that, The method is applied to the central processing unit in the first node; the step of obtaining the first in-card tensor generated by the multiple expert networks that process the first word includes: Perform the following operations via the main thread: Obtain the original in-card tensors output by the multiple expert networks that process the first word; The original layout type of each original card tensor is rearranged into the target layout type to obtain the corresponding first card tensor. The original layout type is a layout type supported by the artificial intelligence chip, and the target layout type is a layout type supported by the central processing unit.
3. The method as described in claim 2, characterized in that, The aggregation of the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word element includes: The second inter-card tensor and the third inter-card tensor are aggregated to obtain a preliminary aggregation result, which corresponds to the target layout type. The target layout type corresponding to the preliminary aggregation result is rearranged into the original layout type by the main thread to obtain the target aggregation result.
4. The method as described in claim 1, characterized in that, The step of aggregating the acquired intra-card tensors to obtain inter-card tensors and sending the inter-card tensors to the second node includes: The first thread aggregates the multiple intra-card tensors to obtain the inter-card tensor. The first inter-card tensor is pushed into the first queue to trigger the second thread to send the first inter-card tensor in the first queue to the second node via remote direct memory access.
5. The method as described in claim 1, characterized in that, The receipt of the second inter-card tensor generated for the second term sent by the second node includes: The third thread uses remote direct memory access to receive the second inter-card tensor sent by the second node.
6. The method as described in claim 5, characterized in that, The step of obtaining the third inter-card tensor generated by the first node for the second word includes: The third thread pushes the second inter-card tensor into the second queue to trigger the fourth thread to obtain the second intra-card tensors generated by multiple expert networks that process the second terminology, and aggregates the obtained multiple second intra-card tensors to obtain the third inter-card tensor.
7. The method as described in claim 6, characterized in that, The aggregation of the second inter-card tensor and the third inter-card tensor to obtain the target aggregation result corresponding to the second word element includes: The second inter-card tensor is saved to shared memory via the third thread; The fourth thread reads the second inter-card tensor from the shared memory and aggregates the second inter-card tensor with the third inter-card tensor to obtain the target aggregation result.
8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1-7.