A tensor distribution method, device, storage medium and program product

Abstract

This application provides a tensor distribution method, device, storage medium, and program product, particularly relating to the field of artificial intelligence chips. The method includes: determining K expert networks matching each first term tensor from a hybrid expert model; constructing a first mapping relationship based on multiple first term tensors and the K expert networks matching each first term tensor; obtaining a second mapping relationship, the second mapping relationship including: multiple second term tensors obtained by a second node and the network identifiers of the K expert networks matching each second term tensor; obtaining a target mapping relationship based on the first and second mapping relationships; and distributing the set of term tensors matching each expert network to the artificial intelligence chip deploying that expert network based on the target mapping relationship, without needing to distribute all term tensors to the artificial intelligence chip deploying the expert network, thereby reducing invalid data transmission, improving bandwidth utilization, and reducing transmission latency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a tensor distribution method, device, storage medium, and program product. Background Technology

[0002] In large language models, Mixture of Experts (MoE) models are widely used to improve model capacity and inference efficiency. These models consist of multiple expert networks. In a MoE model, each token is distributed to multiple expert networks for parallel processing via a router. In distributed inference scenarios, there are situations where the expert networks and the tokens they process reside on different devices. Therefore, communication methods are needed to distribute the tokens processed by each expert network to the devices where those networks reside for further processing.

[0003] Under related technologies, a general all-to-all aggregate communication method is used to distribute tokens to the devices where the expert network resides for processing. In this communication method, each participating device (such as a Graphics Processing Unit (GPU)) sends tokens to all other devices in the cluster and simultaneously receives tokens from all other devices. However, in reality, the expert network on one device may only need to process tokens sent by some other devices, resulting in a large amount of invalid data transmission, leading to low bandwidth utilization and high transmission latency. Summary of the Invention

[0004] This application provides a tensor distribution method, device, storage medium, and program product for reducing invalid data transmission, thereby improving bandwidth utilization and reducing transmission latency.

[0005] On one hand, embodiments of this application provide a tensor distribution method applied to a first node, the method comprising: Obtain multiple first-word tensors; For each first word tensor, determine K expert networks from the hybrid expert model that match the first word tensor, where K is greater than 1; Based on the multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor, a first mapping relationship is constructed. Obtain the second mapping relationship, which includes: multiple second word tensors obtained by the second node, and the network identifiers of the K expert networks that match each second word tensor; Based on the first mapping relationship and the second mapping relationship, the target mapping relationship is obtained; For each expert network, based on the target mapping relationship, the set of lexical tensors that matches the expert network is distributed to the artificial intelligence chip where the expert network is deployed. The set of lexical tensors includes the first lexical tensor and / or the second lexical tensor.

[0006] On one hand, embodiments of this application provide a tensor distribution device applied to a first node, the device comprising: The acquisition module is used to acquire multiple first word tensors; for each first word tensor, K expert networks matching the first word tensor are determined from the hybrid expert model, where K is greater than 1; A construction module is used to construct a first mapping relationship based on the plurality of first word tensors and the network identifiers of the K expert networks that match each first word tensor; obtain a second mapping relationship, the second mapping relationship including: the plurality of second word tensors obtained by the second node and the network identifiers of the K expert networks that match each second word tensor; and obtain a target mapping relationship based on the first mapping relationship and the second mapping relationship. A distribution module is configured to distribute a set of lexical tensors matching the expert network to the artificial intelligence chip on which the expert network is deployed, for each expert network. The set of lexical tensors includes the first lexical tensor and / or the second lexical tensor.

[0007] Optionally, the method is applied to the central processing unit in the first node; The acquisition module is specifically used for: Read the raw lexical tensors sent by each of the multiple artificial intelligence chips from the shared memory of the central processing unit; The original layout type of each original lexical tensor is rearranged into the target layout type to obtain the corresponding first lexical tensor. The original layout type is the layout type supported by the artificial intelligence chip, and the target layout type is the layout type supported by the central processing unit.

[0008] Optionally, the acquisition module is specifically used for: Obtain the weight matrix of the router in the hybrid expert model, and construct a feature matrix based on the multiple first word tensors; Perform matrix multiplication on the feature matrix and the weight matrix to obtain the processing result, which includes the matching degree of each first word tensor with the N expert networks contained in the hybrid expert model, where N is greater than K; For each first word tensor, based on the matching degree between the first word tensor and the N expert networks, select the K matching expert networks from the N expert networks.

[0009] Optionally, the building module is specifically used for: Using a remote direct memory access method, it receives multiple second term tensors sent by the second node, as well as the network identifiers of K expert networks that match each second term tensor. A second mapping relationship is constructed based on the multiple second word tensors and the network identifiers of K expert networks that match each second word tensor.

[0010] Optionally, the building module is specifically used for: The system uses remote direct memory access to receive the second mapping relationship sent by the second node. The second mapping relationship is constructed by the second node based on the plurality of second word tensors and the network identifiers of K expert networks that match each second word tensor.

[0011] Optionally, the building module is specifically used for: Based on the target mapping relationship, obtain each first word tensor that matches the expert network, and each second word tensor that matches the expert network; The first lexical tensor and the second lexical tensor are distributed to the artificial intelligence chip on which the expert network is deployed.

[0012] Optionally, the acquisition module is further configured to: Using a remote direct memory access method, the multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor are sent to the second node.

[0013] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip. When the artificial intelligence chip executes the program, it implements the steps of the tensor distribution method described above.

[0014] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the tensor distribution method described above.

[0015] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the tensor distribution method described above.

[0016] In this embodiment, for each of the multiple first word tensors, K matching expert networks are determined from the hybrid expert model. Then, a first mapping relationship is constructed based on the multiple first word tensors and the network identifiers of the K expert networks matching each first word tensor. A second mapping relationship is obtained, which includes the multiple second word tensors obtained by the second node and the network identifiers of the K expert networks matching each second word tensor. Based on the first and second mapping relationships, a target mapping relationship is obtained. Thus, based on the target mapping relationship, the set of word tensors (including first word tensors and / or second word tensors) that each expert network needs to process can be obtained. Then, for each expert network, the set of word tensors that the expert network needs to process is distributed to the AI ​​chip where the expert network is deployed, instead of distributing all word tensors to the AI ​​chip where the expert network is deployed. This greatly reduces invalid data transmission, thereby improving bandwidth utilization and reducing transmission latency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1A This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application; Figure 1B This application provides a schematic diagram of the structure of a node according to an embodiment of the present application; Figure 2 This is a schematic diagram of a tensor distribution method provided in an embodiment of this application; Figure 3 This is a schematic diagram of another tensor distribution method provided in an embodiment of this application; Figure 4 This is a schematic diagram of a tensor distribution device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0020] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.

[0021] refer to Figure 1A This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least: video memory 101 and computing unit 102. This application does not specifically limit the number of video memory 101 and computing unit 102.

[0022] The computing unit 102 can be a streaming processing cluster (SPC). The video memory 101 can be high-bandwidth memory (HBM) or other types of memory.

[0023] In this embodiment of the application, one or more expert networks from a hybrid expert model can be deployed in the artificial intelligence chip 100.

[0024] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0025] Artificial intelligence chips 100 can be: GPUs, general-purpose computing on graphics processing units (GPGPUs), domain-specific architectures (DSAs), etc.

[0026] refer to Figure 1B This is a schematic diagram of the structure of a node applicable to an embodiment of this application. In actual applications, node 103 can be a server, terminal device, etc.; the node 103 includes a central processing unit 104 and multiple artificial intelligence chips 100.

[0027] In node 103, the central processing unit 104 can communicate with any one of the artificial intelligence chips 100. Of course, any two artificial intelligence chips 100 can also communicate with each other. This application does not make any specific limitations on this.

[0028] In this embodiment of the application, multiple expert networks in the MoE model can be deployed on multiple artificial intelligence chips within a node 103, and one or more expert networks can be deployed on one artificial intelligence chip.

[0029] The MoE model can also be deployed on multiple nodes 103, with each node 103 deploying a portion of the expert network from the MoE model. In this embodiment, the first node can be any one of the multiple nodes 103, and the second node can be any one of the multiple nodes 103 other than the first node. It should be noted that the multiple nodes 103 may include other nodes besides the first and second nodes; however, this application does not specifically limit this.

[0030] The following is based on Figure 1A The diagram shows the architecture of the artificial intelligence chip. Figure 1B The diagram shown illustrates the node architecture and provides a detailed description of a tensor distribution method offered in this application. (See also...) Figure 2 The process of this method is executed by the first node, and the method includes the following steps: Step 201: Obtain multiple first-word tensors.

[0031] Specifically, the tensor distribution method of this application can be applied to various scenarios, such as image processing, speech processing, and text processing. In different application scenarios, the physical meaning of the distributed lexical tensors can be different.

[0032] For example, in a text processing scenario, the distributed lexical tensor can be text data used in tasks such as text generation and text recognition.

[0033] For example, in a speech processing scenario, the distributed lexical tensor can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.

[0034] For example, in image processing scenarios, the distributed term tensors can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0035] In this embodiment, lexical units and lexical tensors represent the forms of the same semantic unit at different stages of the model. The user's input text is segmented into lexical units. These lexical units are then input into a large language model, where one or more network layers map them to corresponding lexical tensors. Finally, the lexical tensors are input into the MoE model within the large language model for further processing.

[0036] The MoE model comprises multiple expert networks. For any given word tensor, multiple expert networks matching the word tensor are identified from the MoE model. Then, a router distributes the word tensor to the matching expert networks for parallel processing, obtaining the processing result (which is also a tensor). This processing result can be output to the next network layer for further processing or output directly.

[0037] In some embodiments, the tensor distribution method of this application can be executed by the central processing unit in the first node. The method reads the original lexical tensors sent by multiple artificial intelligence chips from the shared memory of the central processing unit; rearranges the original layout type of each original lexical tensor into a target layout type to obtain the corresponding first lexical tensor, where the original layout type is a layout type supported by the artificial intelligence chip and the target layout type is a layout type supported by the central processing unit.

[0038] In practice, within the first node, multiple AI chips copy the generated raw lexical tensors to the shared memory of the central processing unit (CPU). The CPU's main thread reads the raw lexical tensors from the shared memory; at this point, the raw layout type of the raw lexical tensors is a layout type supported by the AI ​​chips.

[0039] In order to adapt to the layout types supported by the CPU, the CPU's main thread rearranges the original lexical tensor to obtain the first lexical tensor of the target layout type, that is, the first lexical tensor of the layout types supported by the CPU.

[0040] In this embodiment, the distribution and transmission process of lexical tensors is offloaded to the CPU side, reducing reliance on AI chips. This allows for full utilization of computing and storage resources in heterogeneous computing clusters, reducing the pressure on the GPU memory of AI chips. Secondly, the larger memory capacity and shared memory mechanism of the CPU side enable the distribution of lexical tensors, effectively improving data distribution efficiency. Furthermore, the lexical tensor distribution within the CPU side is implemented on shared memory, allowing for higher performance and not being limited by specific hardware architectures.

[0041] Step 202: For each first word tensor, determine K expert networks from the hybrid expert model that match the first word tensor.

[0042] Specifically, when K is greater than 1, the expert networks in the hybrid expert model can be deployed on one or more nodes. For example, multiple expert networks in the hybrid expert model can be deployed on the first node; or multiple expert networks in the hybrid expert model can be deployed on the first node and the second node, that is, some expert networks in the hybrid expert model are deployed in the first node, and other expert networks in the hybrid expert model are deployed in the second node.

[0043] During the training of the hybrid expert model, different expert networks automatically differentiate into their respective strengths (such as grammar, mathematical reasoning, image texture processing, etc.). In practical applications, the expert network can be a feedforward network or a more complex substructure; this application does not impose any specific limitations on this.

[0044] In this embodiment, K is a preset value, which represents the number of expert networks that process lexical tensors in parallel. The K expert networks can be all or part of the expert networks in a hybrid expert model.

[0045] In some embodiments, the present application selects at least K matching expert networks for each first word tensor in the following manner: Obtain the weight matrix of the router in the hybrid expert model, and construct the feature matrix based on multiple first-term tensors; then perform matrix multiplication on the feature matrix and the weight matrix to obtain the processing result, which includes: the matching degree of each first-term tensor with the N expert networks included in the hybrid expert model, where N is greater than K; for each first-term tensor, select K matching expert networks from the N expert networks based on the matching degree of the first-term tensor with the N expert networks.

[0046] Specifically, the router's weight matrix is ​​obtained during the training of the hybrid expert model, and the shape of the weight matrix is ​​as follows: Where H represents the length of the feature vector of the first word tensor (HiddenDim). This represents the number of expert networks in the hybrid expert model (NumExperts).

[0047] Each first-word tensor is a one-dimensional tensor. Multiple first-word tensors are concatenated to obtain a feature matrix, the shape of which is: ,in, This represents the number of first-order token tensors (NumTokens).

[0048] The main thread of the central processing unit performs matrix multiplication with the feature matrix as the left matrix and the weight matrix as the right matrix to obtain the processing result, where the shape of the processing result is: The processing result is normalized to obtain a score matrix, where the score of each row in the score matrix represents the degree of matching between a first-word tensor and N expert networks.

[0049] For each first word tensor, sort the N expert networks in descending order of matching degree to obtain the sorting result; take the top K expert networks in the sorting result as the K expert networks that match the first word tensor.

[0050] Step 203: Based on multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor, construct the first mapping relationship.

[0051] Specifically, the first mapping relationship can take the form of a table, graph, etc. For different first word tensors, the assigned K expert networks may be different or the same. After matching K expert networks to each first word tensor, the first word tensors that each expert network needs to process can be aggregated to obtain the first mapping relationship; that is, the first mapping relationship includes: the mapping relationship between each expert network and each first word tensor that each expert network needs to process.

[0052] Step 204: Obtain the second mapping relationship, and based on the first and second mapping relationships, obtain the target mapping relationship.

[0053] Specifically, the second mapping relationship includes: multiple second word tensors obtained by the second node, and the network identifiers of the K expert networks that match each second word tensor.

[0054] After the first node matches K expert networks for each first word tensor, it can aggregate the first word tensors that each expert network needs to process to obtain a first mapping relationship; that is, the first mapping relationship indicates the mapping relationship between each expert network and each first word tensor that each expert network needs to process. Similarly, the second mapping relationship indicates the mapping relationship between each expert network and each second word tensor that each expert network needs to process.

[0055] Thus, by combining the first and second mapping relationships to obtain the target mapping relationship, the target mapping relationship indicates the mapping relationship between each expert network and each first lexical tensor and each second lexical tensor that each expert network needs to process.

[0056] In this embodiment of the application, the first node obtains the second mapping relationship in at least the following ways: In the first implementation method, the first node uses remote direct memory access to receive the second mapping relationship sent by the second node. The second mapping relationship is constructed by the second node based on multiple second word tensors and the network identifiers of K expert networks that match each second word tensor.

[0057] Specifically, after the second node matches K expert networks for each second term tensor, the second term tensors that each expert network in the hybrid expert network needs to process are aggregated to obtain the second mapping relationship. Then, the second mapping relationship is sent to the first node using the Remote Direct Memory Access (RDMA) method.

[0058] Implementation Method 2: The first node receives multiple second term tensors sent by the second node, as well as the network identifiers of K expert networks that match each second term tensor, using a remote direct memory access method; then, based on the multiple second term tensors and the network identifiers of the K expert networks that match each second term tensor, a second mapping relationship is constructed.

[0059] Specifically, after receiving multiple second word tensors and the network identifiers of the K expert networks that match each second word tensor, the first node aggregates the second word tensors that each expert network needs to process to obtain the second mapping relationship.

[0060] In implementation method three, the first node uses remote direct memory access to receive multiple second term tensors, network identifiers of N expert networks, and the matching degree between each second term tensor and the N expert networks sent by the second node. For each first term tensor, the first node selects K matching expert networks from the N expert networks based on the matching degree between the second term tensor and the N expert networks; then, based on the multiple second term tensors and the network identifiers of the K expert networks that match each second term tensor, a second mapping relationship is constructed.

[0061] It should be noted that in the implementation of this application, the number of second nodes can be one or more; that is, the first node can receive the second mapping relationship sent by each of the multiple second nodes; or, the first node can construct multiple second mapping relationships based on the data sent by each of the multiple second nodes; and then merge the first mapping relationship and the multiple second mapping relationships to obtain the target mapping relationship.

[0062] In this embodiment, the first node and the second node transmit data via remote direct memory access, which can effectively improve the efficiency of data transmission between nodes and reduce the data transmission latency between nodes.

[0063] Step 205: For each expert network, the set of lexical tensors that match the expert network is distributed to the AI ​​chip where the expert network is deployed, based on the target mapping relationship.

[0064] Specifically, the lexical tensor set includes a first lexical tensor and / or a second lexical tensor.

[0065] If each expert network in the hybrid expert model is deployed on the AI ​​chip within the first node, then within the first node, the central processing unit distributes the lexical tensor set that each expert network needs to process to the AI ​​chip where that expert network is deployed.

[0066] If some expert networks in a hybrid expert model are deployed on the first node and some expert networks are deployed on other nodes, then for each expert network deployed on the first node, the central processing unit distributes the set of lexical tensors that the expert network needs to process to the artificial intelligence chip deployed on the first node.

[0067] For each expert network deployed on other nodes, the central processing unit of the first node can distribute the set of lexical tensors that the expert network needs to process to the artificial intelligence chip that deploys the expert network on other nodes. Alternatively, the central processing unit in other nodes can obtain the target mapping relationship using the method described above, and distribute the set of lexical tensors that the expert network needs to process to the artificial intelligence chip that deploys the expert network locally based on the target mapping relationship.

[0068] In this embodiment, for each of the multiple first word tensors, K matching expert networks are determined from the hybrid expert model. Then, a first mapping relationship is constructed based on the multiple first word tensors and the network identifiers of the K expert networks matching each first word tensor. A second mapping relationship is obtained, which includes the multiple second word tensors obtained by the second node and the network identifiers of the K expert networks matching each second word tensor. Based on the first and second mapping relationships, a target mapping relationship is obtained. Thus, based on the target mapping relationship, the set of word tensors (including first word tensors and / or second word tensors) that each expert network needs to process can be obtained. Then, for each expert network, the set of word tensors that the expert network needs to process is distributed to the AI ​​chip where the expert network is deployed, instead of distributing all word tensors to the AI ​​chip where the expert network is deployed. This greatly reduces invalid data transmission, thereby improving bandwidth utilization and reducing transmission latency.

[0069] In some embodiments, for each expert network, a first term tensor matching the expert network and a second term tensor matching the expert network are obtained based on the target mapping relationship; then the first term tensor and the second term tensor are distributed to the artificial intelligence chip where the expert network is deployed.

[0070] Specifically, since the central processing unit (CPU) and the artificial intelligence (AI) chip support different layout types, before the CPU distributes the lexical tensor to the AI ​​chip where the matching expert network is located, it first rearranges the layout of the lexical tensor through the main thread to obtain the original lexical tensor of the original layout type supported by the AI ​​chip. Then, the original lexical tensor of the original layout type is distributed to the AI ​​chip where the matching expert network is located to ensure the accuracy of subsequent processing by the AI ​​chip.

[0071] For each expert network, when the expert network is deployed in the AI ​​chip of the first node, the central processing unit of the first node rearranges the layout of each first word tensor and each second word tensor that the expert network needs to process, and distributes the rearranged first word tensor and each second word tensor to the AI ​​chip in the first node where the expert network is deployed.

[0072] When the expert network is deployed on the AI ​​chip within the second node, the central processing unit (CPU) of the first node rearranges the layout of each first-term tensor and each second-term tensor that the expert network needs to process. Then, using remote direct memory access (RDBMI), it distributes the rearranged first-term tensors and second-term tensors to the AI ​​chip in the second node where the expert network is deployed. Alternatively, the CPU of the first node can stop distributing the term tensors that the expert network needs to process, and instead, the CPU of the second node distributes these term tensors.

[0073] In this embodiment, the distribution and transmission process of lexical tensors is offloaded to the CPU side, effectively reducing the pressure on the GPU memory of the AI ​​chip. Secondly, the lexical tensor distribution and transmission computation on the CPU side can be executed concurrently with the computation of shared experts on the AI ​​chip side, effectively realizing the concurrent execution of non-dependent modules within the software, thereby reducing overall task latency.

[0074] In some embodiments, a remote direct memory access method is used to send multiple first word tensors and the network identifiers of K expert networks that match each first word tensor to the second node.

[0075] Specifically, the second node constructs a first mapping relationship based on the received multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor. Simultaneously, the second node constructs a second mapping relationship based on the received multiple second word tensors and the network identifiers of the K expert networks that match each second word tensor. The first and second mapping relationships are then combined to construct a target mapping relationship. The process by which the second node obtains the target mapping relationship is the same as that of the first node, and will not be repeated here.

[0076] For each expert network, when the expert network is deployed in the AI ​​chip within the second node, the second node determines the set of lexical tensors (including the first lexical tensor and / or the second lexical tensor) that the expert network needs to process based on the target mapping relationship, and distributes the lexical tensor set to the AI ​​chip where the expert network is deployed.

[0077] When the expert network is deployed in the AI ​​chip within the first node, the second node can distribute the lexical tensor set that the expert network needs to process to the AI ​​chip where the expert network is deployed; alternatively, it can stop distributing the lexical tensor set that the expert network needs to process and have the first node distribute the lexical tensor set that the expert network needs to process. This application does not specifically limit this.

[0078] In this embodiment of the application, in a multi-node scenario, each node can use the CPU in parallel to distribute lexical tensors, thereby improving the overall processing efficiency of the task.

[0079] To better explain the embodiments of this application, the following describes a tensor distribution method provided by the embodiments of this application, taking the central processing unit of the first node as an example. The central processing unit includes multiple threads, namely: main thread, thread one, thread two, thread three, and thread four; see also Figure 3 The tensor distribution method of this application specifically includes the following steps: Step 301: The generated first word tensor is copied from the multiple GPUs within the first node to the shared memory of the central processing unit.

[0080] Step 302: The main thread reads multiple first-term tensors from shared memory.

[0081] Step 303: The main thread performs layout rearrangement on each first word tensor.

[0082] Among them, the multiple first-word tensors after the layout rearrangement constitute the feature matrix.

[0083] Step 304: The main thread performs matrix multiplication on the feature matrix and the router's weight matrix to obtain the matrix multiplication result.

[0084] The matrix multiplication result includes the matching scores of each first word tensor with the N expert networks in the hybrid expert model.

[0085] Step 305: The main thread obtains K expert networks that match each first word tensor based on the matrix multiplication result.

[0086] Step 306: The main thread obtains the first mapping table based on multiple first word tensors and K expert networks that match each first word tensor.

[0087] The first mapping table packet represents the first term tensor that each expert network needs to process.

[0088] Step 307: Thread 1 sends multiple first-term tensors to the second node via RDMA.

[0089] Step 308: Thread 2 sends the network identifiers of the K expert networks that match each first lexical tensor to the second node via RDMA.

[0090] Step 309: Thread 2 sends the matching scores of the K expert networks that match each first word tensor to the second node via RDMA.

[0091] Step 310: Thread 3 receives multiple second-term tensors sent by the second node.

[0092] Step 311: Thread 4 receives the network identifiers of the K expert networks that match each second term tensor sent by the second node.

[0093] Step 312: Thread 4 receives the matching scores from the second node for each second word tensor from the K expert networks.

[0094] Step 313: Thread 4 constructs a second mapping table based on the information received in steps 310-312; and merges the first mapping table and the second mapping table to obtain the target mapping table.

[0095] The second mapping table package represents the second term tensor that each expert network needs to process.

[0096] Step 314: The main thread collects the word tensors that each expert network needs to process according to the target mapping table to obtain the word tensor matrix; and rearranges the layout of each word tensor in the word tensor matrix.

[0097] Step 315: The main thread distributes the lexical tensor matrix of each layout rearrangement to the GPUs that deploy the corresponding expert networks.

[0098] In this embodiment, the distribution and transmission process of lexical tensors is offloaded to the CPU side, reducing reliance on the GPU. This allows for full utilization of the computing and storage resources in the heterogeneous computing cluster, reducing the pressure on the GPU's video memory. Secondly, by utilizing the CPU's larger memory capacity and shared memory mechanism, the distribution and processing of lexical tensors are achieved, effectively improving data distribution efficiency. Furthermore, the lexical tensor distribution and transmission computation on the CPU side can be executed concurrently with the computation of shared experts on the GPU side, effectively realizing the concurrent execution of non-dependent modules within the software, thereby reducing overall task latency.

[0099] Based on the same technical concept, this application provides a schematic diagram of a tensor distribution device, applied to a first node, such as... Figure 4 As shown, the device 400 includes: The acquisition module 401 is used to acquire multiple first word tensors; for each first word tensor, K expert networks matching the first word tensor are determined from the hybrid expert model, where K is greater than 1; The construction module 402 is used to construct a first mapping relationship based on the plurality of first word tensors and the network identifiers of the K expert networks that match each first word tensor; obtain a second mapping relationship, the second mapping relationship including: the plurality of second word tensors obtained by the second node and the network identifiers of the K expert networks that match each second word tensor; and obtain a target mapping relationship based on the first mapping relationship and the second mapping relationship. The distribution module 403 is used to distribute a set of lexical tensors that matches the expert network to the artificial intelligence chip where the expert network is deployed, for each expert network. The set of lexical tensors includes the first lexical tensor and / or the second lexical tensor.

[0100] Optionally, the method is applied to the central processing unit in the first node; The acquisition module 401 is specifically used for: Read the raw lexical tensors sent by each of the multiple artificial intelligence chips from the shared memory of the central processing unit; The original layout type of each original lexical tensor is rearranged into the target layout type to obtain the corresponding first lexical tensor. The original layout type is the layout type supported by the artificial intelligence chip, and the target layout type is the layout type supported by the central processing unit.

[0101] Optionally, the acquisition module 401 is specifically used for: Obtain the weight matrix of the router in the hybrid expert model, and construct a feature matrix based on the multiple first word tensors; Perform matrix multiplication on the feature matrix and the weight matrix to obtain the processing result, which includes the matching degree of each first word tensor with the N expert networks contained in the hybrid expert model, where N is greater than K; For each first word tensor, based on the matching degree between the first word tensor and the N expert networks, select the K matching expert networks from the N expert networks.

[0102] Optionally, the construction module 402 is specifically used for: Using a remote direct memory access method, it receives multiple second term tensors sent by the second node, as well as the network identifiers of K expert networks that match each second term tensor. A second mapping relationship is constructed based on the multiple second word tensors and the network identifiers of K expert networks that match each second word tensor.

[0103] Optionally, the construction module 402 is specifically used for: The system uses remote direct memory access to receive the second mapping relationship sent by the second node. The second mapping relationship is constructed by the second node based on the plurality of second word tensors and the network identifiers of K expert networks that match each second word tensor.

[0104] Optionally, the construction module 402 is specifically used for: Based on the target mapping relationship, obtain each first word tensor that matches the expert network, and each second word tensor that matches the expert network; The first lexical tensor and the second lexical tensor are distributed to the artificial intelligence chip on which the expert network is deployed.

[0105] Optionally, the acquisition module 401 is further configured to: Using a remote direct memory access method, the multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor are sent to the second node.

[0106] In this embodiment, for each of the multiple first word tensors, K matching expert networks are determined from the hybrid expert model. Then, a first mapping relationship is constructed based on the multiple first word tensors and the network identifiers of the K expert networks matching each first word tensor. A second mapping relationship is obtained, which includes the multiple second word tensors obtained by the second node and the network identifiers of the K expert networks matching each second word tensor. Based on the first and second mapping relationships, a target mapping relationship is obtained. Thus, based on the target mapping relationship, the set of word tensors (including first word tensors and / or second word tensors) that each expert network needs to process can be obtained. Then, for each expert network, the set of word tensors that the expert network needs to process is distributed to the AI ​​chip where the expert network is deployed, instead of distributing all word tensors to the AI ​​chip where the expert network is deployed. This greatly reduces invalid data transmission, thereby improving bandwidth utilization and reducing transmission latency.

[0107] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0108] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to at least one artificial intelligence chip. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited. Figure 5 Taking the connection between the AI ​​chip 100 and the memory 501 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0109] In this embodiment of the application, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 501, the at least one artificial intelligence chip 100 can perform the steps of the tensor distribution method described above.

[0110] The artificial intelligence chip 100 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It performs tensor distribution by running or executing instructions stored in the memory 501 and accessing data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units. It may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0111] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0112] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 501 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0113] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the tensor distribution method described above.

[0114] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the tensor distribution method described above.

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0118] These computer program instructions may also be loaded onto a computer device or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer device or other programmable apparatus to produce a process implemented by the computer device, such that the instructions, which execute on the computer device or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0119] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0120] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A tensor distribution method, characterized in that, Applied to the first node, the method includes: Obtain multiple first-word tensors; For each first word tensor, determine K expert networks from the hybrid expert model that match the first word tensor, where K is greater than 1; Based on the multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor, a first mapping relationship is constructed. Obtain the second mapping relationship, which includes: multiple second word tensors obtained by the second node, and the network identifiers of the K expert networks that match each second word tensor; Based on the first mapping relationship and the second mapping relationship, the target mapping relationship is obtained; For each expert network, based on the target mapping relationship, the set of lexical tensors that matches the expert network is distributed to the artificial intelligence chip where the expert network is deployed. The set of lexical tensors includes the first lexical tensor and / or the second lexical tensor.

2. The method as described in claim 1, characterized in that, The method is applied to the central processing unit in the first node; obtaining multiple first word tensors includes: Read the raw lexical tensors sent by each of the multiple artificial intelligence chips from the shared memory of the central processing unit; The original layout type of each original lexical tensor is rearranged into the target layout type to obtain the corresponding first lexical tensor. The original layout type is the layout type supported by the artificial intelligence chip, and the target layout type is the layout type supported by the central processing unit.

3. The method as described in claim 1, characterized in that, For each first word tensor, determining K expert networks from the hybrid expert model that match the first word tensor includes: Obtain the weight matrix of the router in the hybrid expert model, and construct a feature matrix based on the multiple first word tensors; Perform matrix multiplication on the feature matrix and the weight matrix to obtain the processing result, which includes the matching degree of each first word tensor with the N expert networks contained in the hybrid expert model, where N is greater than K; For each first word tensor, based on the matching degree between the first word tensor and the N expert networks, select the K matching expert networks from the N expert networks.

4. The method as described in claim 1, characterized in that, The process of obtaining the second mapping relationship includes: Using a remote direct memory access method, it receives multiple second term tensors sent by the second node, as well as the network identifiers of K expert networks that match each second term tensor. A second mapping relationship is constructed based on the multiple second word tensors and the network identifiers of K expert networks that match each second word tensor.

5. The method as described in claim 1, characterized in that, The process of obtaining the second mapping relationship includes: The system uses remote direct memory access to receive the second mapping relationship sent by the second node. The second mapping relationship is constructed by the second node based on the plurality of second word tensors and the network identifiers of K expert networks that match each second word tensor.

6. The method as described in claim 1, characterized in that, The step of distributing the set of lexical tensors matching the expert network to the AI ​​chip deploying the expert network, based on the target mapping relationship, for each expert network, includes: Based on the target mapping relationship, obtain each first word tensor that matches the expert network, and each second word tensor that matches the expert network; The first lexical tensor and the second lexical tensor are distributed to the artificial intelligence chip on which the expert network is deployed.

7. The method as described in claim 1, characterized in that, Also includes: Using a remote direct memory access method, the multiple first word tensors and the network identifiers of the K expert networks that match each first word tensor are sent to the second node.

8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method according to any one of claims 1-7.

Images