A tensor determination method, device and storage medium based on expert parallelism

Abstract

The embodiment of the present application provides a tensor determination method, device and storage medium based on expert parallelism, which relates to the field of artificial intelligence chip technology. The method includes: splitting the input tensor into N sub-tensors in the hidden layer dimension of the input tensor; and assigning the N sub-tensors to the expert sub-networks deployed in each of the N computing units for parallel calculation. Since each expert sub-network calculates the assigned sub-tensor through the corresponding first weight parameter, the first weight parameter of the expert sub-network and the assigned sub-tensor need to be adapted in shape; based on this, for the target dimension associated with the hidden layer dimension of the sub-tensor in the first weight parameter, the length of the target dimension is set to the length of the hidden layer dimension of the sub-tensor, thereby reducing the length of the first weight parameter in the target dimension, that is, reducing the video memory space required by the weight parameter of the expert sub-network itself, thereby reducing the storage pressure of the video memory.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence chip technology, and in particular to a tensor determination method, device, and storage medium based on expert parallelism. Background Art

[0002] Deep learning network models mainly include two development directions: horizontal and vertical. As the model develops deeper and deeper in the horizontal and vertical directions, the model's demand for video memory of artificial intelligence chips is also increasing.

[0003] To reduce the model's memory usage, related technologies use memory offload to optimize the memory. This involves temporarily storing some memory data in the central processing unit (CPU) to reduce memory usage until the next time the data is needed, at which point it is copied from the CPU to the memory.

[0004] However, copying large amounts of data between the CPU and the AI ​​chip is an extremely time-consuming process, which will result in significant performance loss for the entire model. Summary of the Invention

[0005] The embodiments of the present application provide a tensor determination method, device, and storage medium based on expert parallelism, which are used to reduce the model's occupancy of video memory, reduce data copying between the CPU and the artificial intelligence chip, and thus improve the overall performance of the model.

[0006] In one aspect, an embodiment of the present application provides a tensor determination method based on expert parallelism, the method comprising:

[0007] Split the input tensor into N sub-tensors in the hidden layer dimension of the input tensor, where N is greater than 1;

[0008] Allocate the N sub-tensors to expert sub-networks deployed in respective N computing units; each expert sub-network includes a corresponding first weight parameter, and the shape of each first weight parameter includes a target dimension associated with the hidden layer dimension; the length of the sub-tensor allocated to each expert sub-network in the hidden layer dimension is equal to the length of the first weight parameter of each expert sub-network in the target dimension;

[0009] By using N expert sub-networks, the N sub-tensors are calculated in parallel based on respective first weight parameters to obtain corresponding sub-calculation results;

[0010] Based on the obtained N sub-computation results, an output tensor is obtained.

[0011] In one aspect, an embodiment of the present application provides a tensor determination device based on expert parallelism, the device comprising:

[0012] A splitting module is used to split the input tensor into N sub-tensors in the hidden layer dimension of the input tensor, where N is greater than 1;

[0013] an allocation module, configured to allocate the N sub-tensors to expert sub-networks deployed in respective N computing units; each expert sub-network includes a corresponding first weight parameter, and the shape of each first weight parameter includes a target dimension associated with the hidden layer dimension; the length of the sub-tensor allocated to each expert sub-network in the hidden layer dimension is equal to the length of the first weight parameter of each expert sub-network in the target dimension;

[0014] The execution module is used to calculate the N sub-tensors in parallel based on their respective first weight parameters through N expert sub-networks to obtain corresponding sub-calculation results; and obtain an output tensor based on the obtained N sub-calculation results.

[0015] Optionally, the N computing units are located in M ​​artificial intelligence chips, where M is greater than 0 and less than or equal to N;

[0016] The first weight parameter of the expert sub-network deployed by each computing unit is stored in the video memory of the artificial intelligence chip where each computing unit is located.

[0017] Optionally, the execution module is further configured to:

[0018] When the remaining storage space of the video memory does not meet the storage condition of the first weight parameter, part of the data stored in the video memory is transferred to the central processing unit.

[0019] Optionally, the first weight parameter of the expert sub-network includes: an upper projection layer parameter and a lower projection layer parameter;

[0020] The target dimension in the shape of the upper projection layer parameter is: row dimension;

[0021] The target dimension in the shape of the lower projection layer parameter is: column dimension.

[0022] Optionally, the execution module is further configured to:

[0023] The length of the first weight parameter of each expert sub-network in the target dimension is updated by load balancing loss until the first weight parameters of the N expert sub-networks meet a preset balancing condition.

[0024] Optionally, the execution module is further configured to:

[0025] Through each computing unit, the following operations are performed:

[0026] Collect N first weight parameters to obtain a first global parameter;

[0027] Splitting the first global parameter into N second weight parameters, and distributing the N second weight parameters to other computing units; the N second weight parameters have the same length in the target dimension;

[0028] Copying an assigned second weight parameter to the distributed optimizer for parameter update to obtain a third weight parameter;

[0029] Collect N third weight parameters to obtain a second global parameter, and obtain a fourth weight parameter from the second global parameter, where the fourth weight parameter has the same length in the target dimension as the first weight parameter of the expert subnetwork deployed by the computing unit.

[0030] Optionally, the execution module is further configured to:

[0031] The allocated second weight parameter is copied to the distributed optimizer for parameter update. Before obtaining the third weight parameter, the second weight parameter of the original precision is adjusted to the second weight parameter of the target precision, and the target precision is greater than the original precision.

[0032] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned expert parallel-based tensor determination method.

[0033] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned expert parallel-based tensor determination method.

[0034] In an embodiment of the present application, the input tensor is split into N sub-tensors in the hidden layer dimension of the input tensor; the N sub-tensors are assigned to the expert sub-networks deployed in each of the N computing units for parallel calculation. Since each expert sub-network calculates the assigned sub-tensor using the corresponding first weight parameter, the first weight parameter of the expert sub-network and the assigned sub-tensor need to be adapted in shape; based on this, for the target dimension associated with the hidden layer dimension of the sub-tensor in the first weight parameter, the length of the target dimension is set to the length of the hidden layer dimension of the sub-tensor. In this way, while ensuring the accurate calculation of the sub-tensor, the length of the first weight parameter in the target dimension is reduced, that is, the video memory space required by the weight parameter of the expert sub-network itself is reduced.

[0035] Since the expert subnetwork's weight parameters occupy less memory, the amount of memory data copied between the CPU and the AI ​​chip is reduced, thereby improving the efficiency of model training and inference, and ultimately, the overall model performance. Furthermore, since the total number of expert subnetwork weight parameters is reduced, the amount of data required for data calculation and communication is also reduced, thereby improving the performance of the entire hybrid expert network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a tensor determination method based on expert parallelism provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a hybrid expert model provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a weight parameter updating method provided in an embodiment of the present application Figure 1 ;

[0041] Figure 5 A schematic diagram of a weight parameter updating method provided in an embodiment of the present application Figure 2 ;

[0042] Figure 6 A schematic diagram of the structure of a tensor determination device based on expert parallelism provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] refer to Figure 1, which is a structural diagram of an artificial intelligence chip applicable to embodiments of the present application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102. The computing units 102 may be streaming processing clusters (SPCs). Video memory 101 may be high-bandwidth memory (HBM) or other types of memory.

[0046] In the embodiment of the present application, the hybrid expert network includes multiple expert sub-networks, each of which is deployed in a computing unit 102, and the first weight parameter of each expert sub-network is stored in the video memory 101. The multiple expert sub-networks can be deployed in the computing unit 102 included in a single artificial intelligence chip 100, or in the computing units 102 included in multiple artificial intelligence chips 100, and this application does not impose any specific restrictions on this.

[0047] When performing expert parallel tensor determination, the input tensor is split into N sub-tensors in its hidden layer dimension. These N sub-tensors are then assigned to the expert sub-networks deployed in each of the N computing units 102 for parallel computation. Because each expert sub-network computes the assigned sub-tensor using a corresponding first weight parameter, the first weight parameter of the expert sub-network and the assigned sub-tensor must be adapted in shape. Based on this, the length of the target dimension associated with the sub-tensor's hidden layer dimension in the first weight parameter is set to the length of the sub-tensor's hidden layer dimension.

[0048] Compared to conventional solutions (where the length of the first weight parameter in the target dimension is equal to the length of the input tensor in the hidden layer dimension), this application reduces the length of the first weight parameter in the target dimension, thereby reducing the amount of video memory space required by the expert sub-network's weight parameters. Consequently, when the first weight parameters of each expert sub-network are stored in video memory 101, the storage space occupied by the first weight parameters in video memory 101 is significantly reduced, thereby alleviating storage pressure on video memory 101.

[0049] Since the expert sub-network's weight parameters occupy less video memory, the amount of video memory data copied between the CPU and the AI ​​chip 100 can be reduced, thereby improving the efficiency of model training and inference. Furthermore, since the total number of expert sub-network weight parameters is reduced, the amount of data required for data calculation and data communication by the computing unit 102 is correspondingly reduced, thereby improving the overall performance of the AI ​​chip 100.

[0050] In addition to the above-mentioned structure, the artificial intelligence chip 100 in this application may also include other structures, which are not specifically limited in this application.

[0051] The artificial intelligence chip 100 can be: a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain-specific architecture (DSA), etc.

[0052] The following is based on Figure 1 The architecture diagram of the artificial intelligence chip shown in the figure provides a detailed introduction to the process of tensor determination method based on expert parallelism, see Figure 2 The method is executed by a computer device, the computer device comprising Figure 1 The artificial intelligence chip shown in FIG. 1 includes the following steps:

[0053] Step 201: Split the input tensor into N sub-tensors in the hidden layer dimension of the input tensor.

[0054] Specifically, N is greater than 1. The expert parallel-based tensor determination method can be applied to various scenarios, such as image processing, speech processing, and text processing. In different application scenarios, the physical meaning of the input tensor can be different.

[0055] For example, in a text processing scenario, the input tensor can be text data used in tasks such as text generation and text recognition.

[0056] For example, in a speech processing scenario, the input tensor can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.

[0057] For example, in an image processing scenario, the input tensor can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0058] The shape of the input tensor consists of one or more dimensions. When the input tensor is a two-dimensional tensor, the shape of the input tensor consists of: the sequence dimension and the hidden layer dimension.

[0059] For example, consider a text processing scenario where we take the original text "I love NLP." We split the original text into a sequence of three tokens: "I," "love," and "NLP." We then perform an encoding operation on each token in the sequence to obtain a corresponding feature vector of length 1024.

[0060] The tag sequence and the feature vector corresponding to each tag constitute the input tensor, where the tag sequence corresponds to the sequence dimension, and the length of the sequence dimension is 3; the feature vector corresponds to the hidden layer dimension, and the length of the hidden layer dimension is 1024.

[0061] In the sequence dimension and hidden layer dimension, the sequence dimension is the row dimension and the hidden layer dimension is the column dimension. Taking the text processing scenario as an example, the size of the input tensor corresponding to the original text "I love NLP" is: .

[0062] The CPU splits the input tensor into N sub-tensors in the hidden layer dimension of the input tensor, where the lengths of the N sub-tensors in the hidden layer dimension can be the same or different; the lengths of the N sub-tensors in the sequence dimension are the same.

[0063] Step 202: distribute the N sub-tensors to the expert sub-networks deployed in the N computing units.

[0064] Specifically, the hybrid expert network includes multiple expert sub-networks. In the embodiment of the present application, the hybrid expert network can be an independent model or a sub-network in a deep learning model.

[0065] For example, the deep learning model is set to include 16 layers in the vertical dimension, and is split into 4 groups of networks according to pipeline parallelism (PP). Each group of networks is deployed in an artificial intelligence chip, and each group of networks includes 4 layers.

[0066] For example, see Figure 3 The four layers in a network are: a self-attention mechanism, a residual connection (Add) + layer normalization layer, a feed-forward network (FFN) layer, and a residual connection + layer normalization layer. The feed-forward network layer uses a hybrid expert network architecture, consisting of four expert sub-networks (FFN1, FFN2, FFN3, and FFN4), a router, and a merging unit. The router distributes the four sub-tensors to the four expert sub-networks for parallel computation (i.e., expert parallelism), while the merging unit combines the computational results of the four expert sub-networks into a single final output.

[0067] Furthermore, data parallelism (DP) and expert parallelism can share multiple AI chips. Specifically, when using both data and expert parallelism in a deep learning model, expert parallelism reuses the AI ​​chips allocated to data parallelism. In other words, data parallelism is enabled first, while expert parallelism is disabled, allowing multiple AI chips to be used for data parallelism. When the model reaches the feedforward neural network layer, data parallelism is disabled and expert parallelism is enabled, allowing multiple AI chips to be used for expert parallelism.

[0068] It's important to note that deep learning models can employ tensor parallelism (TP) in addition to data parallelism. Similarly, expert parallelism can reuse AI chips assigned to tensor parallelism. When a deep learning model employs data parallelism, tensor parallelism, and expert parallelism simultaneously, multiple AI chips are assigned to data parallelism and tensor parallelism, and expert parallelism can reuse AI chips assigned to either type of parallelism. This not only reduces the amount of video memory used by each AI chip, but also improves the overall performance of the deep learning model.

[0069] In an embodiment of the present application, each expert subnetwork is deployed in a computing unit, and each expert subnetwork includes a corresponding first weight parameter, which is a two-dimensional tensor. Since each expert subnetwork calculates the assigned subtensor using the corresponding first weight parameter, the shape of each first weight parameter includes a target dimension associated with the hidden layer dimension; the length of the subtensor assigned to each expert subnetwork in the hidden layer dimension is equal to the length of the first weight parameter of each expert subnetwork in the target dimension. In addition to including the target dimension, the shape of the first weight parameter also includes a reference dimension, the length of which is determined by the type of the expert subnetwork.

[0070] For example, the hybrid expert network includes 4 expert sub-networks, and the size of the input tensor is: , where the length of the hidden layer dimension is 8192. The CPU splits the input tensor into 4 sub-tensors in the hidden layer dimension, and the size of each sub-tensor is: The CPU assigns each sub-tensor to an expert sub-network deployed on a computational unit.

[0071] The first weight parameter of each expert sub-network is a two-dimensional tensor, and the size of the first weight parameter is: When performing matrix multiplication with the sub-tensor as the left matrix and the first weight parameter as the right matrix, the length of the sub-tensor column dimension (i.e., the hidden layer dimension) is equal to the row dimension (i.e., the target dimension) of the first weight parameter, and both are 2048.

[0072] In some implementations, N computing units are located in M ​​artificial intelligence chips, where M is greater than 0 and less than or equal to N; the first weight parameter of the expert subnetwork deployed by each computing unit is stored in the video memory of the artificial intelligence chip where each computing unit is located.

[0073] Specifically, the number of expert sub-networks deployed on each artificial intelligence chip is the same, and corresponding storage space is allocated in the video memory for the first weight parameter of each expert sub-network.

[0074] For example, a hybrid expert network consists of eight expert subnetworks deployed on four AI chips, with two expert subnetworks deployed on each chip. For each AI chip, two expert subnetworks are deployed on two computing units, respectively, and the first weight parameters of each expert subnetwork are stored in the AI ​​chip's video memory.

[0075] Step 203 : Using N expert sub-networks, N sub-tensors are calculated in parallel based on their respective first weight parameters to obtain corresponding sub-calculation results.

[0076] Step 204: Obtain an output tensor based on the obtained N sub-computation results.

[0077] Specifically, in each computation unit, a matrix multiplication is performed on the input tensor and the first weight parameter of the expert sub-network to obtain the corresponding sub-computation result, which can be a two-dimensional tensor. The N sub-computation results are then concatenated to obtain an output tensor, where the size of the output tensor is the same as the size of the input tensor.

[0078] In an embodiment of the present application, the input tensor is split into N sub-tensors in the hidden layer dimension of the input tensor; the N sub-tensors are assigned to the expert sub-networks deployed in each of the N computing units for parallel calculation. Since each expert sub-network calculates the assigned sub-tensor using the corresponding first weight parameter, the first weight parameter of the expert sub-network and the assigned sub-tensor need to be adapted in shape; based on this, for the target dimension associated with the hidden layer dimension of the sub-tensor in the first weight parameter, the length of the target dimension is set to the length of the hidden layer dimension of the sub-tensor. In this way, while ensuring the accurate calculation of the sub-tensor, the length of the first weight parameter in the target dimension is reduced, that is, the video memory space required by the weight parameter of the expert sub-network itself is reduced.

[0079] Since the expert subnetwork's weight parameters occupy less memory, the amount of memory data copied between the CPU and the AI ​​chip is reduced, thereby improving the efficiency of model training and inference, and ultimately, the overall model performance. Furthermore, since the total number of expert subnetwork weight parameters is reduced, the amount of data required for data calculation and communication is also reduced, thereby improving the performance of the entire hybrid expert network.

[0080] In some implementations, the first weight parameters of the expert subnetwork include: an upper projection layer parameter and a lower projection layer parameter; the target dimension in the shape of the upper projection layer parameter is: a row dimension; and the target dimension in the shape of the lower projection layer parameter is: a column dimension.

[0081] Specifically, the upper projection layer parameters and the lower projection layer parameters are both two-dimensional tensors. A matrix multiplication is performed on the input tensor and the upper projection layer parameters to obtain an upper projection result, where the size of the upper projection result is larger than the size of the input tensor. A matrix multiplication is performed on the upper projection result and the lower projection layer parameters to obtain a sub-calculation result, where the size of the sub-calculation result is the same as the size of the input tensor.

[0082] For example, assume that the hybrid expert network includes 8 expert sub-networks and the size of the input tensor is: , that is, the length of the sequence dimension is 2048 and the length of the hidden layer dimension is 8192.

[0083] In the traditional hybrid expert network, the input tensor is split into 8 sub-tensors in the sequence dimension, and the size of each sub-tensor is: , Then, the size of the up-projection layer parameters of the hybrid expert network is: , the size of the lower projection layer parameters is: , where the parameter values ​​"28672" and "14336" are determined by the type of hybrid expert network. That is, for each expert sub-network, the memory needs to store a size of The upper projection layer parameters and size are The lower projection layer parameters, which consumes a lot of video memory.

[0084] In this embodiment of the present application, the input tensor is split into 8 sub-tensors in the hidden layer dimension, and the size of each sub-tensor obtained is: , Then, the size of the up-projection layer parameters of the hybrid expert network is: , the size of the lower projection layer parameters is: , where "28672" and "14336" are determined by the type of hybrid expert network. That is, for each expert sub-network, the memory needs to store a size of The upper projection layer parameters and size are The lower projection layer parameters.

[0085] Clearly, compared to traditional splitting methods, the method in this embodiment significantly reduces the size of the upper and lower projection layer parameters, effectively reducing the memory load on each AI chip and alleviating parameter memory pressure. Furthermore, since the total number of parameters is reduced, the computational effort required for the corresponding parameters is also reduced, which in turn improves the performance of the entire hybrid expert network structure.

[0086] In addition, in the current era of big data, the input features may be ever-changing, including voice, text, images, coordinates, and so on. In order to obtain an ideal result by inputting rich and diverse features into the model, in actual operation, it is necessary to first map each feature to a unique feature vector, and at the same time, prune according to the similarity between the feature vectors and the correlation between the feature vectors and the model accuracy, so as to obtain a set of feature vectors with higher accuracy and higher performance for model training. With reference to the above background, the hidden layer dimension of the present application can be the feature vector dimension of the label sequence. The segmentation based on the hidden layer dimension in the present application can be understood as the segmentation of large-scale independent feature vectors. At the same time, different expert sub-networks are selected for different label sequences from the hybrid expert network, which is improved to selecting different expert sub-networks for different feature vectors. Therefore, the solution of the present application has strong theoretical applicability.

[0087] The technical solution of the embodiment of the present application is applicable to both the model training stage and the model application stage. In the model training stage, it can reduce the video memory requirements and video memory peak value during the hybrid expert network training process, while improving the overall performance of the hybrid expert network.

[0088] During the model training process, not only the specific value of the first weight parameter is adjusted, but also the shape of the first weight parameter is adjusted, that is, the length of the first weight parameter in the target dimension is adjusted.

[0089] In some implementations, at the beginning of model training, the first weight parameters of each expert sub-network are in an unbalanced state, that is, the first weight parameters of some expert sub-networks have a large length in the target dimension, and these weight parameters occupy a large amount of video memory space; while the first weight parameters of other expert sub-networks have a small length in the target dimension, and these weight parameters occupy a very small amount of video memory space. This situation results in insufficient utilization of the video memory.

[0090] In view of this, the present application updates the length of the first weight parameter of each expert subnetwork in the target dimension through load balancing loss until the first weight parameters of the N expert subnetworks meet the preset balancing condition.

[0091] Among them, the balancing condition can be that the lengths of the first weight parameters of multiple expert sub-networks in the target dimension are the same, or that the difference in the lengths of the first weight parameters of any two expert sub-networks in the target dimension is within a preset threshold; of course, the balancing condition can also be in other forms, and this application does not make specific limitations on this.

[0092] Taking the upper projection layer parameter in the first weight parameter as an example, the hybrid expert network includes N expert sub-networks, where N is greater than 1. At the beginning of training, the size of the upper projection layer parameter of each expert sub-network x is: ,in, , the upper projection layer parameters of the N expert sub-networks have different lengths in the target dimension, and the difference is large. After multiple rounds of iterative training, the upper projection layer parameters of the N expert sub-networks meet the preset equilibrium conditions. That is, when the equilibrium state is reached, the size of each upper projection layer parameter is: , that is, the N up-projection layer parameters have the same length in the target dimension.

[0093] The hybrid expert network is gradually transitioned from an unbalanced state to a balanced state through load balancing loss, so that the memory space occupied by the first weight parameters of each expert sub-network is gradually balanced, thereby improving the memory utilization and the overall performance of the hybrid expert network.

[0094] In some embodiments, at the beginning of model training, the first weight parameters of each expert sub-network are in an unbalanced state, which may cause the weight parameters of some expert sub-networks to occupy too much video memory, or even cause the video memory to be unable to meet storage requirements.

[0095] Based on this, in an embodiment of the present application, when the remaining storage space of the video memory does not meet the storage condition of the first weight parameter, part of the data stored in the video memory is transferred to the central processing unit.

[0096] In a specific implementation, the storage condition may be that the storage space required to be occupied by the first weight parameter is greater than the remaining storage space of the video memory; or the ratio between the storage space required to be occupied by the first weight parameter and the remaining storage space of the video memory is greater than a preset ratio; of course, it may also be other set conditions, and this application does not make specific limitations on this.

[0097] During the initial training phase, some data stored in video memory is transferred to the CPU. This data can be intermediate results from model training or inference. This frees up more memory space to store the weight parameters of the expert subnetwork, ensuring model training performance. When the hybrid expert network transitions from an unbalanced to a balanced state, less data is transferred from video memory to the CPU, reducing memory data copies between the CPU and AI chip, thereby improving overall model performance.

[0098] In some embodiments, during model training, a distributed optimizer is often used to update the first weight parameters of each expert sub-network. In practical applications, the distributed optimizer is a static component, that is, it is used to update the weight parameters of a fixed shape.

[0099] However, in the implementation of this application, the model training process requires adjusting the shape of the first weight parameter, that is, adjusting the length of the first weight parameter in the target dimension. In other words, during the training process, the shape of the first weight parameter changes dynamically, which makes it difficult for the distributed optimizer to adapt to the shape of the first weight parameter that needs to be updated.

[0100] In view of this, in the embodiments of this application, see Figure 4 , through each computing unit, perform the following operations respectively:

[0101] Step 401: Collect N first weight parameters to obtain a first global parameter.

[0102] Specifically, through the allgather operation, the first weight parameters sent by the other N-1 computing units are received, and then the first weight parameters of the local expert sub-network are spliced ​​with the received N-1 first weight parameters in a first splicing order to obtain the first global parameter, wherein the first splicing order determines the position of each first weight parameter in the first global parameter.

[0103] Step 402: Split the first global parameter into N second weight parameters, and distribute the N second weight parameters to other computing units.

[0104] Specifically, a scatter operation is used to evenly split the first global parameter into N second weight parameters along its target dimension, and these N second weight parameters are distributed to other computational units. The N second weight parameters have the same length along the target dimension; in other words, they have the same size.

[0105] Step 403: Copy the allocated second weight parameter to the distributed optimizer for parameter update to obtain a third weight parameter.

[0106] Specifically, each computing unit corresponds to a distributed optimizer, and each distributed optimizer is used to update the second weight parameter allocated to the corresponding computing unit, wherein the shape of the second weight parameter is always fixed.

[0107] In some implementations, since the large model is trained with low precision and a high-precision parameter is retained when the parameters are updated at the distributed optimizer, in an embodiment of the present application, the second weight parameter of the original precision is adjusted to the second weight parameter of the target precision, and the target precision is greater than the original precision; then the second weight parameter of the target precision is copied to the distributed optimizer for parameter update to obtain a third weight parameter, wherein the third weight parameter corresponds to the target precision.

[0108] In other words, the dynamics of the low-precision weight parameter (i.e., the first weight parameter) is retained, and the statics of the distributed optimizer during parameter update are maintained with the help of the high-precision parameter (i.e., the second weight parameter). In this way, the model training process can not only adjust the shape of the first weight parameter, but also indirectly update the parameter value of the first weight parameter with the help of the distributed optimizer.

[0109] Step 404 : Collect N third weight parameters, obtain the second global parameter, and obtain the fourth weight parameter from the second global parameter.

[0110] Specifically, through all-to-all communication, the third weight parameters sent by the other N-1 computing units are received. Then, the locally updated third weight parameters are concatenated with the received N-1 third weight parameters according to a pre-second concatenation order to obtain the second global parameter. The second concatenation order determines the position of each third weight parameter in the second global parameter, and the second concatenation order can be the same as the first concatenation order described above.

[0111] The computing unit obtains a fourth weight parameter from the second global parameter based on the length of the first weight parameter of the locally deployed expert sub-network in the target dimension and the first splicing order. The size of the fourth weight parameter is the same as that of the first weight parameter. The fourth weight parameter is actually the first weight parameter after the parameter is updated.

[0112] At this time, the fourth weight parameter corresponds to the target accuracy, that is, corresponds to high precision; in order to facilitate subsequent model training, the fourth weight parameter of the target accuracy is adjusted to the fourth weight parameter of the original accuracy.

[0113] For example, see Figure 5 , the hybrid expert network includes N expert sub-networks, and the N expert sub-networks are deployed in N computing units. The N computing units are: The size of the first weight parameter of the expert sub-network deployed by EP1 is: , the precision is BF16 (i.e. 16-bit floating point number); the size of the first weight parameter of the expert sub-network deployed by EP2 is: , the precision is BF16; and so on, EP N The size of the first weight parameter of the deployed expert sub-network is: , the precision is BF16 (i.e. 16-bit floating point number), where .

[0114] Through the full aggregation operation, N first weight parameters are collected and concatenated to obtain the first global parameter, which is: , with an accuracy of BF16.

[0115] The first global parameter is switched to N second weight parameters on average through the scattering operation, and the N second weight parameters are distributed to the other N-1 computing units. The size of the second weight parameter allocated to each computing unit is: , with an accuracy of BF16.

[0116] Adjust the precision of the second weight parameter to FP32 (i.e. 32-bit floating point number), then copy the second weight parameter to the distributed optimizer for parameter update to obtain the third weight parameter. The size of the third weight parameter is: , the precision is FP32.

[0117] Through full exchange communication, the fourth weight parameter of the expert sub-network deployed by each computing unit is obtained. Specifically, the size of the fourth weight parameter of the expert sub-network deployed by EP1 is: , the accuracy is BF16; the size of the fourth weight parameter of the expert sub-network deployed by EP2 is: , the precision is BF16; and so on, EP N The size of the fourth weight parameter of the deployed expert sub-network is: , with an accuracy of BF16.

[0118] In this embodiment, a first weight parameter with a dynamically changing shape is converted into a second weight parameter with a fixed shape through a full gather and scatter operation. The second weight parameter is then added to a distributed optimizer for parameter update, obtaining a third weight parameter. The updated first weight parameter (i.e., the fourth weight parameter) is then restored using the obtained multiple third weight parameters. This allows the first weight parameter to be indirectly updated using the distributed optimizer, thereby ensuring model training effectiveness while significantly reducing the graphics memory space occupied by the first weight parameter.

[0119] Based on the same technical concept, the embodiment of the present application provides a structural diagram of a tensor determination device based on expert parallelism, such as Figure 6 As shown, the expert parallel-based tensor determination device 600 includes:

[0120] A splitting module 601 is configured to split the input tensor into N sub-tensors in the hidden layer dimension of the input tensor, where N is greater than 1;

[0121] an allocating module 602 configured to allocate the N sub-tensors to expert sub-networks deployed in respective N computing units; each expert sub-network includes a corresponding first weight parameter, and the shape of each first weight parameter includes a target dimension associated with the hidden layer dimension; the length of the sub-tensor allocated to each expert sub-network in the hidden layer dimension is equal to the length of the first weight parameter of each expert sub-network in the target dimension;

[0122] The execution module 603 is used to calculate the N sub-tensors in parallel based on the respective first weight parameters through N expert sub-networks to obtain corresponding sub-calculation results; and obtain an output tensor based on the obtained N sub-calculation results.

[0123] Optionally, the N computing units are located in M ​​artificial intelligence chips, where M is greater than 0 and less than or equal to N;

[0124] The first weight parameter of the expert sub-network deployed by each computing unit is stored in the video memory of the artificial intelligence chip where each computing unit is located.

[0125] Optionally, the execution module 603 is further configured to:

[0126] When the remaining storage space of the video memory does not meet the storage condition of the first weight parameter, part of the data stored in the video memory is transferred to the central processing unit.

[0127] Optionally, the first weight parameter of the expert sub-network includes: an upper projection layer parameter and a lower projection layer parameter;

[0128] The target dimension in the shape of the upper projection layer parameter is: row dimension;

[0129] The target dimension in the shape of the lower projection layer parameter is: column dimension.

[0130] Optionally, the execution module 603 is further configured to:

[0131] The length of the first weight parameter of each expert sub-network in the target dimension is updated by load balancing loss until the first weight parameters of the N expert sub-networks meet a preset balancing condition.

[0132] Optionally, the execution module 603 is further configured to:

[0133] Through each computing unit, the following operations are performed:

[0134] Collect N first weight parameters to obtain a first global parameter;

[0135] Splitting the first global parameter into N second weight parameters, and distributing the N second weight parameters to other computing units; the N second weight parameters have the same length in the target dimension;

[0136] Copying an assigned second weight parameter to the distributed optimizer for parameter update to obtain a third weight parameter;

[0137] Collect N third weight parameters to obtain a second global parameter, and obtain a fourth weight parameter from the second global parameter, where the fourth weight parameter has the same length in the target dimension as the first weight parameter of the expert subnetwork deployed by the computing unit.

[0138] Optionally, the execution module 603 is further configured to:

[0139] The allocated second weight parameter is copied to the distributed optimizer for parameter update. Before obtaining the third weight parameter, the second weight parameter of the original precision is adjusted to the second weight parameter of the target precision, and the target precision is greater than the original precision.

[0140] In an embodiment of the present application, the input tensor is split into N sub-tensors in the hidden layer dimension of the input tensor; the N sub-tensors are assigned to the expert sub-networks deployed in each of the N computing units for parallel calculation. Since each expert sub-network calculates the assigned sub-tensor using the corresponding first weight parameter, the first weight parameter of the expert sub-network and the assigned sub-tensor need to be adapted in shape; based on this, for the target dimension associated with the hidden layer dimension of the sub-tensor in the first weight parameter, the length of the target dimension is set to the length of the hidden layer dimension of the sub-tensor. In this way, while ensuring the accurate calculation of the sub-tensor, the length of the first weight parameter in the target dimension is reduced, that is, the video memory space required by the weight parameter of the expert sub-network itself is reduced.

[0141] Since the expert subnetwork's weight parameters occupy less memory, the amount of memory data copied between the CPU and the AI ​​chip is reduced, thereby improving the efficiency of model training and inference, and ultimately, the overall model performance. Furthermore, since the total number of expert subnetwork weight parameters is reduced, the amount of data required for data calculation and communication is also reduced, thereby improving the performance of the entire hybrid expert network.

[0142] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0143] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 7 As shown, it includes at least one artificial intelligence chip 100 and a memory 701 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 701 is not limited in the embodiment of the present application. Figure 7 For example, the artificial intelligence chip 100 and the memory 701 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0144] In an embodiment of the present application, the memory 701 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 701, the at least one artificial intelligence chip 100 can perform the steps of the above-mentioned expert parallel-based tensor determination method.

[0145] Among them, the artificial intelligence chip 100 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device. By running or executing instructions stored in the memory 701 and calling data stored in the memory 701, it can realize expert parallel tensor determination. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 701 can be implemented on the same chip. In some embodiments, they can also be implemented separately on independent chips.

[0146] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in the processor.

[0147] The memory 701 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The memory 701 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic storage device, a magnetic disk, an optical disk, and the like. The memory 701 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 701 in the embodiments of the present application may also be a circuit or any other device capable of performing a storage function, used to store program instructions and / or data.

[0148] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned expert parallel-based tensor determination method.

[0149] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned expert parallel-based tensor determination method.

[0150] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer readable memory that can direct a computer device or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operating steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0154] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0155] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A tensor determination method based on expert parallelism, characterized in that: include: In the hidden layer dimension of the input tensor, the input tensor is split into N sub-tensors, where N is greater than 1; the input tensor is any one of the following: text data, voice data, and image data; Allocate the N sub-tensors to expert sub-networks deployed in respective N computing units; each of the expert sub-networks includes a corresponding first weight parameter, and the shape of each first weight parameter includes a target dimension associated with the hidden layer dimension; the length of the sub-tensor allocated to each expert sub-network in the hidden layer dimension is equal to the length of the first weight parameter of each expert sub-network in the target dimension; By using N expert sub-networks, the N sub-tensors are calculated in parallel based on respective first weight parameters to obtain corresponding sub-calculation results; Based on the obtained N sub-calculation results, an output tensor is obtained; During the model training process, the length of the first weight parameter of each expert subnetwork in the target dimension is updated through the load balancing loss until the difference in the length of the first weight parameters of any two expert subnetworks in the target dimension is within a preset threshold.

2. The method according to claim 1, wherein The N computing units are located in M ​​artificial intelligence chips, where M is greater than 0 and less than or equal to N; The first weight parameter of the expert sub-network deployed by each computing unit is stored in the video memory of the artificial intelligence chip where each computing unit is located.

3. The method according to claim 2, wherein Also includes: When the remaining storage space of the video memory does not meet the storage condition of the first weight parameter, part of the data stored in the video memory is transferred to the central processing unit.

4. The method according to claim 1, wherein The first weight parameters of the expert sub-network include: upper projection layer parameters and lower projection layer parameters; The target dimension in the shape of the upper projection layer parameter is: row dimension; The target dimension in the shape of the lower projection layer parameter is: column dimension.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Through each computing unit, the following operations are performed: Collect N first weight parameters to obtain a first global parameter; Splitting the first global parameter into N second weight parameters, and distributing the N second weight parameters to other computing units; the N second weight parameters have the same length in the target dimension; Copying an assigned second weight parameter to the distributed optimizer for parameter update to obtain a third weight parameter; Collect N third weight parameters to obtain a second global parameter, and obtain a fourth weight parameter from the second global parameter, where the fourth weight parameter has the same length in the target dimension as the first weight parameter of the expert subnetwork deployed by the computing unit.

6. The method according to claim 5, wherein The step of copying the allocated second weight parameter to the distributed optimizer for parameter update and obtaining the third weight parameter further includes: The second weight parameter of the original precision is adjusted to the second weight parameter of the target precision, where the target precision is greater than the original precision.

7. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that It stores a computer program executed by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to perform the steps of the method according to any one of claims 1 to 6.

Images