Batch sequence hybrid parallel method, device, storage medium and program product
By using a batch-sequence hybrid parallel approach, the problem of hardware difficulty in supporting the target storage format is solved, achieving more efficient computation and memory utilization, and improving chip performance and versatility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2026-02-03
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies, after replacing the storage format of input tensors from batch and sequence dimensions, make it difficult for hardware to support the target storage format, resulting in a decrease in chip performance.
By employing a batch-sequence hybrid parallel method without replacing the batch and sequence dimensions, the target tensor is obtained and uniformly divided into M sub-tensors, which are then distributed across M devices for parallel computation, thereby reducing memory usage and improving computational efficiency.
It improves the overall performance of the chip, reduces the hardware memory requirements, and increases computing efficiency, while also having better versatility and compatibility.
Smart Images
Figure CN122133729A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a batch-sequence hybrid parallel method, device, storage medium, and program product. Background Technology
[0002] Deep learning network models mainly have two development directions: horizontal and vertical. As models become deeper in both directions, their demand for GPU memory on AI chips also increases. Training large models typically uses multi-parallel techniques such as tensor parallelism and sequence parallelism to reduce the memory consumption of a single GPU. Sequence parallelism refers to partitioning the input tensor along a sequence dimension and distributing the partitioned results across multiple AI chips for parallel computation.
[0003] To achieve sequence parallelism, related techniques typically modify the original storage format of the input tensor [batch, sequence, hidden] to the target storage format [sequence, batch, hidden], that is, replace the batch dimension and the sequence dimension; then the input tensor is split from the sequence dimension, where batch represents the batch dimension, sequence represents the sequence dimension, and hidden represents the hidden layer dimension.
[0004] Since the original storage format is a storage format supported by the hardware (i.e., the AI chip), after modifying the original storage format of the input tensor to the target storage format, it becomes difficult for the hardware to support processing the input tensor in the target storage format, thus affecting the chip's performance. Summary of the Invention
[0005] This application provides a batch-sequence hybrid parallel method, device, storage medium, and program product, which can achieve the effect of batch-sequence hybrid parallelism without replacing the batch dimension and sequence dimension, thereby improving the overall performance of the chip.
[0006] On the one hand, embodiments of this application provide a batch-sequence hybrid parallel method, the method comprising: Obtain the original batch size and original sequence length of the original tensor; When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor; wherein, the value of the sequence parallelism is M, and the target batch size of the target tensor is an integer multiple of M, where M is greater than 1; The target tensor is uniformly divided into M first sub-tensors from the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result.
[0007] On one hand, embodiments of this application provide a batch-sequence mixed parallel apparatus, the apparatus comprising: The acquisition module is used to obtain the original batch size and original sequence length of the original tensor; A transformation module is used to perform a shape transformation operation on the original tensor to obtain a target tensor when the ratio of the original batch size to the sequence parallelism is not an integer; wherein the value of the sequence parallelism is M, the target batch size of the target tensor is an integer multiple of M, and M is greater than 1; The segmentation module is used to uniformly segment the target tensor into M first sub-tensors from a batch dimension, and distribute the M first sub-tensors to M devices for parallel computation to obtain a first computation result.
[0008] Optionally, the segmentation module is specifically used for: Assign the M first sub-tensors to the M devices; The first type of operator is executed in parallel on the allocated first sub-tensor through the M devices to obtain the corresponding sub-computation result; The obtained M sub-calculation results are aggregated to obtain the first calculation result.
[0009] Optionally, the first type of operator includes: a layer normalization operator and a regularization operator; the batch size of the first calculation result is the target batch size, and the sequence length of the first calculation result is the target sequence length.
[0010] Optionally, the segmentation module is further configured to: When the ratio of the original batch size to the sequence parallelism is an integer, the original tensor is uniformly divided into M second sub-tensors from the batch dimension; The M second sub-tensors are distributed to the M devices for parallel computation to obtain the second computation result.
[0011] Optionally, the segmentation module is further configured to: The M first sub-tensors are distributed to M devices for parallel computation. After obtaining the first computation result, the weight tensor is evenly divided into M third sub-tensors, and the M third sub-tensors are distributed to the M devices. The third calculation result is obtained by performing the second type of operator on the first calculation result and the allocated third sub-tensor in parallel using the M devices.
[0012] Optionally, the second type of operator is an attention mechanism operator; The segmentation module is specifically used for: For each device, an intermediate tensor is generated by the device based on the first calculation result and the third sub-tensor; The device extracts query tensor, key tensor, and value tensor from the intermediate tensor, and executes the attention mechanism operator based on the query tensor, key tensor, and value tensor to obtain the corresponding attention calculation results. The M attention calculation results are aggregated to obtain the third calculation result.
[0013] Optionally, the batch size of the third calculation result is the target batch size, and the sequence length of the third calculation result is the target sequence length.
[0014] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip. When the artificial intelligence chip executes the computer program, it implements the steps of the above-described batch sequence hybrid parallel method.
[0015] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the batch sequence hybrid parallel method described above.
[0016] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described batch sequence hybrid parallel method.
[0017] In this embodiment, firstly, the original batch size and original sequence length of the original tensor are obtained. When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor, such that the target batch size of the target tensor in the batch dimension is an integer multiple of M (the value of the sequence parallelism). This allows the target tensor to be uniformly divided into M first sub-tensors in the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result. During the computation process, each device does not need to store the entire target tensor, but only needs to store the allocated first sub-tensors, which greatly reduces the GPU memory usage. Simultaneously, when M devices perform parallel computation on the target tensor, computational efficiency is greatly improved, thereby enhancing the overall performance of model training.
[0018] Secondly, this application does not require the replacement of the batch and sequence dimensions of the tensor, which is more hardware-friendly, requires less of the underlying hardware, and has better versatility. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application; Figure 2 A flowchart illustrating a batch-sequence hybrid parallel method provided in an embodiment of this application; Figure 3 A flowchart illustrating an attention mechanism calculation method provided in an embodiment of this application; Figure 4 A flowchart illustrating a batch-sequence hybrid parallel and tensor parallel method provided in an embodiment of this application; Figure 5 A schematic diagram of a batch sequence hybrid parallel device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0021] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0022] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be streaming processing clusters (SPCs). The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.
[0023] In this embodiment, the original batch size and original sequence length of the original tensor are obtained; when the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor, such that the target batch size of the target tensor in the batch dimension is an integer multiple of the sequence parallelism. In this way, the target tensor can be evenly divided into M first sub-tensors in the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result.
[0024] Among them, the M devices can be M artificial intelligence chips 100, M computing units 102, or other types of devices. This application does not make specific limitations in this regard.
[0025] In some embodiments, the number M of devices pre-configured to perform sequential parallel computing is obtained, for example, the number of AI chips 100 used to perform sequential parallel computing; and then the number M of devices is used as the value of the degree of sequential parallelism.
[0026] When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor. Then, the target tensor is evenly divided into M first sub-tensors from the batch dimension. These M first sub-tensors are then distributed across M AI chips 100 for parallel computation. In this way, the GPU memory 101 of each AI chip 100 does not need to store the entire target tensor, but only the allocated first sub-tensors, significantly reducing the space occupied by the GPU memory 101. Simultaneously, when the M AI chips 100 perform parallel computation on the target tensor, they fully utilize the computing resources of all M AI chips 100, greatly improving computational efficiency and thus enhancing the overall performance of model training.
[0027] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.
[0028] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-Purpose Graphics Processing Units (GPGPU), Domain Specific Architecture (DSA), etc.
[0029] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip, and details the process of a batch-sequence hybrid parallel method. (See also...) Figure 2 This method is performed by a computer device, which includes... Figure 1 The artificial intelligence chip shown includes the following steps: Step 201: Obtain the original batch size and original sequence length of the original tensor.
[0030] Specifically, the batch-sequence hybrid parallel method of this application can be applied to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the original tensor can differ in different application scenarios.
[0031] For example, in text processing scenarios, the original tensor can be the text data used in tasks such as text generation and text recognition.
[0032] For example, in a speech processing scenario, the raw tensor can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.
[0033] For example, in image processing scenarios, the original tensor can be image data used in tasks such as image preprocessing, image segmentation, and object detection.
[0034] The shape of the original tensor includes one or more dimensions. When the original tensor is a three-dimensional tensor (i.e., a 3D matrix format), the shape of the original tensor can be represented as [b, s, h], where b represents the original batch size; s represents the original sequence length; and h represents the size of the hidden layer dimension.
[0035] The original batch size of the original tensor refers to the size of the original tensor in the batch dimension, i.e., the number of samples in one training iteration. The original sequence length of the original tensor refers to the size of the original tensor in the sequence dimension, i.e., the length of the sample sequence. The size of the original tensor in the hidden layer dimension is the hidden layer size of the model.
[0036] Step 202: When the ratio of the original batch size to the sequence parallelism is not an integer, perform a shape transformation operation on the original tensor to obtain the target tensor.
[0037] Specifically, the value of sequence parallelism is M; sequence parallelism refers to how many devices the original tensor is divided onto for parallel computation, that is, the number of artificial intelligence chips or computing units that perform parallel computation; sequence parallelism is a preset positive integer greater than 1.
[0038] When the ratio of the original batch size to the sequence parallelism is not an integer, it indicates that the original tensor cannot be evenly divided onto M devices for parallel computation. Therefore, a shape transformation operation (reshape) is performed on the original tensor to obtain the target tensor. The shape transformation operation refers to reorganizing the dimensional structure of the tensor without changing the number and order of elements in the tensor.
[0039] In this embodiment of the application, the target batch size of the target tensor is an integer multiple of M; at the same time, the target batch size of the target tensor is N times the original batch size, and the original sequence length is N times the target sequence length of the target tensor; N also represents the number of parts into which the original tensor is divided in the sequence dimension.
[0040] Specifically, the number of parts N that the original tensor is divided into in the sequence dimension is obtained using the following formula (1): N = LCM(b, M) / b (1) Where LCM represents the least common multiple, b represents the original batch size, and M represents the sequence parallelism value.
[0041] Perform a shape transformation on the original tensor [b, s, h] to obtain the target tensor [b]. [N, s / N, h], where b N is the target batch size, and s / N is the target sequence length.
[0042] Step 203: Divide the target tensor into M first sub-tensors in a batch manner, and distribute the M first sub-tensors to M devices for parallel computation to obtain the first computation result.
[0043] Specifically, the target tensor will be uniformly divided into M first sub-tensors from the batch dimension, and the shape of each first sub-tensor is [b N / M, s / N, h].
[0044] The M first sub-tensors are distributed to M devices for parallel computation, with each device completing the computation of one first sub-tensor; then the sub-computation results obtained by each of the M devices are aggregated (allgather) to obtain the first computation result.
[0045] Among them, M devices can be M artificial intelligence chips, M computing units, or M other types of devices. This application does not make specific limitations in this regard.
[0046] In this embodiment, firstly, the original batch size and original sequence length of the original tensor are obtained. When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor, such that the target batch size of the target tensor in the batch dimension is an integer multiple of the sequence parallelism. This allows the target tensor to be uniformly divided into M first sub-tensors in the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result. For each device, it is not necessary to store the entire target tensor, but only the allocated first sub-tensors, which greatly reduces the GPU memory usage. Simultaneously, when M devices perform parallel computation on the target tensor, computational efficiency is greatly improved, thereby enhancing the overall performance of model training.
[0047] Secondly, this application does not require the replacement of the batch and sequence dimensions of the tensor, which is more hardware-friendly, requires less of the underlying hardware, and has better versatility.
[0048] In some embodiments, M first sub-tensors are assigned to M devices; then, the first type of operator is executed on the assigned first sub-tensors in parallel through the M devices to obtain the corresponding sub-computation results; then, the obtained M sub-computation results are aggregated to obtain the first computation result.
[0049] Specifically, the first type of operator can be an operator in the model that does not support tensor parallelism. Tensor parallelism refers to: splitting the model's weight tensor along a specific dimension (such as the first dimension of the weight tensor), distributing the splitting results to multiple devices for parallel computation, and merging the results after the computation is completed.
[0050] The first category of operators includes the layer normalization operator and the dropout operator. For the layer normalization operator, when performing layer normalization calculation on the input tensor, the complete weight tensor is required, that is, the weight tensor cannot be split, so tensor parallelism cannot be used.
[0051] For regularization operators, random numbers are typically generated independently for each element, and then the elements are discarded using these random numbers; this process does not involve weight tensors, so tensor parallelism cannot be used.
[0052] In order to improve the execution efficiency of the first type of operator and reduce the memory usage of the data related to the first type of operator, this application adopts a batch-sequence hybrid parallel method to split the target tensor into M first sub-tensors and distribute the M first sub-tensors to M devices.
[0053] In this way, for each device, it only needs to store the allocated first sub-tensor in the GPU memory; at the same time, it only needs to execute the first type of operator on the allocated first sub-tensor to obtain the corresponding sub-computation result, which effectively reduces the GPU memory usage of each device; at the same time, the M devices perform the computation on the M first sub-tensors in parallel, which improves the operator execution efficiency and thus improves the overall performance of model training.
[0054] In some embodiments, the shape of the first subtensor is [b [N / M, s / N, h], each device is assigned a first sub-tensor to perform a first-type operator to obtain the corresponding sub-computation result, which has the shape [b N / M, s / N, h].
[0055] The M sub-calculation results obtained from the calculations of the M devices are aggregated to obtain the first calculation result, which has the shape of [b]. [N, s / N, h]; That is, the batch size of the first calculation result is the target batch size, and the sequence length of the first calculation result is the target sequence length.
[0056] In some embodiments, when the ratio of the original batch size to the sequence parallelism is an integer, the original tensor is uniformly divided into M second sub-tensors from the batch dimension; at this time, the shape of each second sub-tensor is [b / M, s, h]. The M second sub-tensors are distributed to M devices for parallel computation to obtain a second computation result.
[0057] For each device, a second sub-tensor is allocated and stored in the video memory; simultaneously, a first type of operator is executed on the allocated second sub-tensor to obtain the corresponding sub-computation result; the shape of the sub-computation result is [b / M, s, h]. The M sub-computation results obtained from the M devices are then aggregated to obtain the second computation result; the shape of the second computation result is [b, s, h].
[0058] In this embodiment, the original tensor is uniformly divided into M second sub-tensors from the batch dimension, and the M second sub-tensors are distributed to M devices for parallel computation. This greatly reduces the computational load and memory usage of each device, thereby improving the operator execution efficiency and thus improving the model training performance.
[0059] In some embodiments, M first sub-tensors are distributed to M devices for parallel computation to obtain a first computation result. Then, the weight tensor is evenly divided into M third sub-tensors, and the M third sub-tensors are distributed to M devices. The second type of operator is then executed on the first computation result and the distributed third sub-tensors in parallel on the M devices to obtain a third computation result.
[0060] Specifically, the second type of operators are those that support tensor parallelism. Examples include attention operators and multilayer perceptron (MLP) operators.
[0061] In practical applications, the weight tensor and tensor parallelism are first obtained, where the tensor parallelism is equal to the sequence parallelism, i.e., the value of the tensor parallelism is M. The weight tensor is uniformly divided into M third sub-tensors from the first dimension, and the M third sub-tensors are distributed to M devices; at the same time, the intermediate tensors are scheduled to M devices respectively.
[0062] For each device, the device stores the allocated third sub-tensor and the first calculation result in the video memory; at the same time, it executes the second type of operator on the allocated third sub-tensor and the first calculation result to obtain the corresponding sub-calculation result; then it aggregates the M sub-calculation results obtained by the M devices to obtain the third calculation result.
[0063] In some embodiments, when the second type of operator is an attention mechanism operator, for each device, an intermediate tensor is generated based on the first calculation result and the third sub-tensor; a query tensor, a key tensor, and a value tensor are extracted from the intermediate tensor, and an attention mechanism operator is executed based on the query tensor, the key tensor, and the value tensor to obtain the corresponding attention calculation result; finally, the M attention calculation results are aggregated to obtain the third calculation result.
[0064] Specifically, see Figure 3 Attention mechanism operators include matrix multiplication operator, first fusion operator, computation operator and second fusion operator.
[0065] First, perform matrix multiplication on the first calculation result and the third sub-tensor to obtain an intermediate tensor with the shape [b [N, s / N, h1]; That is, the batch size of the intermediate tensor is the target batch size, and the sequence length of the intermediate tensor is the target sequence length.
[0066] The first fusion operator has the functions of reshaping, splitting, and permute, used to split the query tensor, key tensor, and value tensor from the intermediate tensor. Since the attention mechanism requires the full sequence length for computation, the sequence dimension cannot be split; and the shape of the intermediate tensor is [b [N, s / N, h1], that is, the sequence dimension is divided into N parts. Therefore, the first fusion operator is also used to adjust the sequence length of the intermediate tensor to the original sequence length s.
[0067] Specifically, after inputting the intermediate tensor into the first fusion operator, a shape transformation operation is first performed on the intermediate tensor to change its shape from [b The intermediate tensor [N, s / N, h1] is adjusted to [b, s, h1], that is, the batch size of the intermediate tensor is adjusted to the original batch size, and the sequence length of the intermediate tensor is adjusted to the original sequence length.
[0068] Next, the intermediate tensor is split to obtain the query tensor q[b, s, n_h]. h2], Key tensor k[b, s, n_g] h2] sum tensor v[b, s, n_g] h2], where n_h is the number of heads in the multi-head attention and n_g is the number of attention groups.
[0069] Next, the dimension permutation operation is performed on the query tensor, key tensor, and value tensor respectively, changing the shape of the query tensor q from [b, s, n_h] to [[b, s, n_h]]. [h2] is adjusted to [b, n_h, s, h2]; the shape of the key tensor k is changed from [b, s, n_g] to [b, s, n_h, h2]. [h2] is adjusted to [b, n_g, s, h2]; the shape of the value tensor v is changed from [b, s, n_g] to [b, n_g]. [h2] is adjusted to [b, n_g, s, h2].
[0070] Next, shape transformation operations are performed on the query tensor, key tensor, and value tensor respectively, adjusting the shape of the query tensor q from [b, n_h, s, h2] to [b [n_h, s, h2]; Adjust the shape of the key tensor k from [b, n_g, s, h2] to [b [n_g, s, h2]; Adjust the shape of the value tensor v from [b, n_g, s, h2] to [b n_g, s, h2].
[0071] Then, input the query tensor, key tensor, and value tensor into the computation operator to perform the attention mechanism calculation, and obtain the attention calculation result. The shape of the attention calculation result is [b n_h, s, h2].
[0072] The second fusion operator has the functions of shape transformation (reshape) and permute dimension, which is used to adjust the sequence length of the sequence dimension of the attention calculation result from s to s / N.
[0073] Specifically, a shape transformation operation is performed on the attention calculation result, changing the shape of the attention calculation result from [b [n_h, s, h2] is adjusted to [b, n_h, s, h2].
[0074] Next, a dimension permutation operation is performed on the attention calculation result, adjusting the shape of the attention calculation result from [b, n_h, s, h2] to [b, s, n_h, h2].
[0075] Next, a shape transformation operation is performed on the attention calculation result, adjusting the shape of the attention calculation result from [b, s, n_h, h2] to [b N, s / N, n_h h2).
[0076] Then, the M attention calculation results obtained from the M devices are aggregated to obtain a third calculation result, the shape of which is [b N, s / N, n_h h2], that is, the batch size of the third calculation result is the target batch size, and the sequence length of the third calculation result is the target sequence length.
[0077] In this embodiment of the application, for the second type of operator that supports tensor parallelism, the weight tensor is evenly divided into M third sub-tensors, and the M third sub-tensors are distributed to M devices for parallel computation. This greatly reduces the amount of computation and memory usage of each device, thereby improving the operator execution efficiency and thus improving the model training performance.
[0078] In some embodiments, the model may include both a first type of operator that does not support tensor parallelism and a second type of operator that does support tensor parallelism; in this case, the first type of operator in the model is computed in a sequential parallel manner (i.e., the batch-sequence hybrid parallelism of this application); the second type of operator in the model is computed in a tensor parallel manner.
[0079] For example, see Figure 4 The model is defined as including: a first-layer normalization operator, an attention mechanism operator, a first regularization operator, a second-layer normalization operator, a multilayer perceptron operator, and a second regularization operator; the shape of the target tensor A1 is defined as [b N, s / N, h].
[0080] The first-level normalization operator is executed using a sequence parallel (i.e., the batch-sequence hybrid parallel method of this application) method, specifically including: uniformly dividing the target tensor A1 into M first sub-tensors from the batch dimension, each first sub-tensor having a shape of [b N / M, s / N, h]. The M first sub-tensors are distributed to M devices. Each device inputs the first sub-tensor into the first-layer normalization operator for computation, obtaining the sub-computation result C1 [b]. N / M, s / N, h].
[0081] Aggregate the M sub-calculation results C1 obtained from the calculations of M devices to obtain the first calculation result A2 [b N, s / N, h].
[0082] The attention mechanism operator is executed in a tensor-parallel manner, specifically including: uniformly dividing the weight tensor into M third sub-tensors, and distributing the M third sub-tensors to M devices; and executing the attention mechanism operator in parallel on the first calculation result A2 and the distributed third sub-tensors through the M devices to obtain the third calculation result A3[b N, s / N, h].
[0083] Perform a reduce_scatter operation on the third calculation result A3 to obtain M fourth sub-tensors B2[b N / M, s / N, h].
[0084] Next, the first regularization operator, the second-level normalization operator, and the aggregation operation are executed sequentially to obtain the fourth calculation result A4[b [N, s / N, h], where the first regularization operator and the second-level normalization operator are executed in a sequential parallel manner. The specific calculation process is the same as the sequential parallel process described above, and will not be repeated here.
[0085] Next, the multilayer perceptron operator is executed using tensor parallelism to obtain the fifth calculation result A5[b The specific calculation process for [N, s / N, h] is the same as the tensor parallel process described above, and will not be repeated here. A reduction and dispersion operation is performed on the fifth calculation result A5 to obtain M fifth sub-tensors B3[b N / M, s / N, h].
[0086] Finally, the second regularization operator and aggregation operation are executed sequentially to obtain the sixth calculation result A6[b]. [N, s / N, h], where the second regularization operator is executed in a sequential parallel manner. The specific calculation process is the same as the sequential parallel process described above, and will not be repeated here.
[0087] In this embodiment, when the model includes a first type of operator and a second type of operator, the first type of operator is executed in sequence parallelism, and the second type of operator is executed in tensor parallelism. That is, both sequence parallelism and tensor parallelism are used simultaneously during model training to maximize parallel computation, greatly reduce the memory usage of each device, and improve the training efficiency of the model. Secondly, this application has fewer requirements for the underlying hardware and has better versatility.
[0088] Based on the same technical concept, this application provides a schematic diagram of the structure of a batch-sequence hybrid parallel device, such as... Figure 5 As shown, the batch sequence mixing parallel device 500 includes: The acquisition module 501 is used to acquire the original batch size and the original sequence length of the original tensor; The transformation module 502 is used to perform a shape transformation operation on the original tensor to obtain a target tensor when the ratio of the original batch size to the sequence parallelism is not an integer; wherein the value of the sequence parallelism is M, the target batch size of the target tensor is an integer multiple of M, and M is greater than 1; The segmentation module 503 is used to uniformly segment the target tensor into M first sub-tensors from a batch dimension, and distribute the M first sub-tensors to M devices for parallel computation to obtain a first computation result.
[0089] Optionally, the segmentation module 503 is specifically used for: Assign the M first sub-tensors to the M devices; The first type of operator is executed in parallel on the allocated first sub-tensor through the M devices to obtain the corresponding sub-computation result; The obtained M sub-calculation results are aggregated to obtain the first calculation result.
[0090] Optionally, the first type of operator includes: a layer normalization operator and a regularization operator; the batch size of the first calculation result is the target batch size, and the sequence length of the first calculation result is the target sequence length.
[0091] Optionally, the segmentation module 503 is further configured to: When the ratio of the original batch size to the sequence parallelism is an integer, the original tensor is uniformly divided into M second sub-tensors from the batch dimension; The M second sub-tensors are distributed to the M devices for parallel computation to obtain the second computation result.
[0092] Optionally, the segmentation module 503 is further configured to: The M first sub-tensors are distributed to M devices for parallel computation. After obtaining the first computation result, the weight tensor is evenly divided into M third sub-tensors, and the M third sub-tensors are distributed to the M devices. The third calculation result is obtained by performing the second type of operator on the first calculation result and the allocated third sub-tensor in parallel using the M devices.
[0093] Optionally, the second type of operator is an attention mechanism operator; The segmentation module 503 is specifically used for: For each device, an intermediate tensor is generated by the device based on the first calculation result and the third sub-tensor; The device extracts query tensor, key tensor, and value tensor from the intermediate tensor, and executes the attention mechanism operator based on the query tensor, key tensor, and value tensor to obtain the corresponding attention calculation results. The M attention calculation results are aggregated to obtain the third calculation result.
[0094] Optionally, the batch size of the third calculation result is the target batch size, and the sequence length of the third calculation result is the target sequence length.
[0095] In this embodiment, firstly, the original batch size and original sequence length of the original tensor are obtained. When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor, such that the target batch size of the target tensor in the batch dimension is an integer multiple of the sequence parallelism. This allows the target tensor to be uniformly divided into M first sub-tensors in the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result. During the computation process, each device does not need to store the entire target tensor, but only needs to store the allocated first sub-tensors, which greatly reduces GPU memory usage. Simultaneously, when M devices perform parallel computation on the target tensor, computational efficiency is greatly improved, thereby enhancing the overall performance of model training.
[0096] Secondly, this application does not require the replacement of the batch and sequence dimensions of the tensor, which is more hardware-friendly, requires less of the underlying hardware, and has better versatility.
[0097] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0098] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 6 As shown, it includes at least one artificial intelligence chip 100 and a memory 601 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 601 is not limited. Figure 6 Taking the connection between the AI chip 100 and the memory 601 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.
[0099] In this embodiment, the memory 601 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 601, the at least one artificial intelligence chip 100 can perform the steps of the batch sequence hybrid parallel method described above.
[0100] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and achieves batch-sequence mixed parallelism by running or executing instructions stored in the memory 601 and retrieving data stored in the memory 601. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 601 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.
[0101] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0102] Memory 601, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 601 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 601 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 601 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0103] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described batch sequence hybrid parallel method.
[0104] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the above-described batch sequence hybrid parallel method.
[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0108] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0109] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A batch-sequence hybrid parallel method, characterized in that, include: Obtain the original batch size and original sequence length of the original tensor; When the ratio of the original batch size to the sequence parallelism is not an integer, a shape transformation operation is performed on the original tensor to obtain the target tensor; wherein, the value of the sequence parallelism is M, and the target batch size of the target tensor is an integer multiple of M, where M is greater than 1; The target tensor is uniformly divided into M first sub-tensors from the batch dimension, and the M first sub-tensors are distributed to M devices for parallel computation to obtain the first computation result.
2. The method as described in claim 1, characterized in that, The step of distributing the M first sub-tensors to M devices for parallel computation to obtain a first computation result includes: Assign the M first sub-tensors to the M devices; The first type of operator is executed in parallel on the allocated first sub-tensor through the M devices to obtain the corresponding sub-computation result; The obtained M sub-calculation results are aggregated to obtain the first calculation result.
3. The method as described in claim 2, characterized in that, The first type of operator includes: layer normalization operator and regularization operator; the batch size of the first calculation result is the target batch size, and the sequence length of the first calculation result is the target sequence length.
4. The method as described in claim 1, characterized in that, Also includes: When the ratio of the original batch size to the sequence parallelism is an integer, the original tensor is uniformly divided into M second sub-tensors from the batch dimension; The M second sub-tensors are distributed to the M devices for parallel computation to obtain the second computation result.
5. The method according to any one of claims 1 to 4, characterized in that, After distributing the M first sub-tensors to M devices for parallel computation and obtaining the first computation result, the method further includes: The weight tensor is uniformly divided into M third sub-tensors, and the M third sub-tensors are distributed to the M devices; The third calculation result is obtained by performing the second type of operator on the first calculation result and the allocated third sub-tensor in parallel using the M devices.
6. The method as described in claim 5, characterized in that, The second type of operator is an attention mechanism operator; the step of performing the second type of operator on the first calculation result and the allocated third sub-tensor in parallel through the M devices to obtain the third calculation result includes: For each device, an intermediate tensor is generated by the device based on the first calculation result and the third sub-tensor; The device extracts query tensor, key tensor, and value tensor from the intermediate tensor, and executes the attention mechanism operator based on the query tensor, key tensor, and value tensor to obtain the corresponding attention calculation results. The M attention calculation results are aggregated to obtain the third calculation result.
7. The method as described in claim 5, characterized in that, The batch size of the third calculation result is the target batch size, and the sequence length of the third calculation result is the target sequence length.
8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1-7.