Generative language model distributed coding system and method

Abstract

The invention discloses a generative language model distributed coding method and system, which comprises a plurality of devices, data can be transmitted between any two devices, and a generative language model is deployed on each device; the generative language model comprises a plurality of repeated attention modules, and each attention module is divided into three parts including the first part, calculation and output of the cache K and the cache V; in the second part, the cache K and the cache V are stored; in the third part, matrix multiplication is carried out on the cache K and Q of Q / K / V dense, and the matrix multiplication is output to a Softmax operator; wherein when the second part of the cache K and the cache V is stored, the cache K and the cache V are broadcasted at the same time, and the value of the cache K and the value of the cache V are stored in parallel equipment. According to the invention, when the large language model is coded on a plurality of devices, the input is distributed as uniformly as possible.

Description

Technical Field

[0001] The present invention belongs to a new generation of information technology, especially to the field of coding, and specifically to a generative language model distributed coding system and method. Background Art

[0002] The attention mechanism is one of the main techniques used in generative language models. Its structure is roughly as follows Figure 1 As shown in the figure, the shape of the operator in the attention mechanism in a large language model changes with the number of tokens inferred by the language model. For example, at the beginning, the number of tokens input by the user is N (the number of tokens generated after the prompt word input by the user is encoded). Figure 1 The shape of the Linear operator (linear layer of the neural network) corresponding to V, K, and Q is N. When a token is inferred, the next round of inference begins, and the number of input tokens becomes N+1 (after each inference, the model will generate a new token id). The shape of the Linear operator in the corresponding model also becomes N+1. This reasoning method is called full reasoning.

[0003] The incremental attention mechanism is a performance optimization technology for the attention mechanism in large language models. In the above reasoning process, the next round of reasoning after the first token does not need to input N+1 tokens, but only inputs 1 token, which is the token in the previous round of reasoning. At this time, the shape of the Linear changes from N+1 to 1, and the shape of any subsequent round of reasoning is always 1. This optimization makes the calculation amount of the Linear always the calculation amount of shape 1 in iterations other than the first round of reasoning, rather than the incremental calculation amount of N+1. However, when the Linear is output to the Scaled dot-productAttention module, it is necessary to merge the output content of the Linear during the previous reasoning, that is, the input shape of the Scaled dot-productAttention is still N+1. Therefore, when the incremental attention mechanism is reasoning, there will be an input cache with a dynamic shape, that is, the input content of the previous round of Scaled dot-product Attention module, the merge operator with incremental shape, and the Scaled dot-product Attention module with dynamic shape.

[0004] Combination Figure 2, Parallel acceleration methods include: data parallel, model parallel, tensorparallel, pipeline parallel. Usually data parallel and pipeline parallel cannot reduce the latency of the same data, but can only improve the overall throughput. Therefore, most generative language models currently use tensor parallelism to reduce latency. At the same time, tensor parallelism can also reduce the storage requirements of the model on a single device (GPU) (light blue part) (For the incremental attention mechanism, the input for the first reasoning is N. When the user enters a longer prompt word, there will be a lot of converted token ids, which makes the first reasoning very time-consuming. We now divide the long token id into M parts, and each token id will become the original N / M. Then we take each token id to different cards to infer the model, which can play a role in accelerating reasoning).

[0005] Tensor parallelism in the attention mechanism is mainly for Linear operators (or dense operators), which occupy more than 90% of the model time. For example, after parallelization on two GPU cards, only 1 / 2 of the storage on each GPU card is allocated to store 1 / 2 weights, and the amount of calculation is reduced to 1 / 2, so the calculation time becomes 1 / 2 of the original, such as Figure 3 As shown in the figure, the additional overhead is one broadcast (hxccl.broadcast) and one collection (hxccl.gather) between cards. The bandwidth of inter-card communication directly affects the speedup ratio of this method.

[0006] Figure 4 Ring Self-Attention is given. Ring Self-Attention transmits the calculation results of QK and output. The main purpose is to reduce the storage of K and V cache on each device. However, the transmission volume between devices is still relatively large, and there is data dependency between the front and back operators between devices, which cannot reduce the delay.

[0007] Figure 5A simplified diagram of a block in the large language model Llama 2 is given. The red color indicates the amount of inter-card transmission of 256 tokens input at a time in a block when using Tensor parallel. Each Dense needs to be broadcast between devices first, and then the gather synchronization is performed after the dense is calculated. The time is the sum of the broadcast time + 1 / n dense time + gather time. When the number of tokens increases, the broadcast+gather time will increase, resulting in multiple devices not being able to reduce the latency to 1 / n. Summary of the invention

[0008] In view of the problems existing in the background technology, the present invention proposes a generative language model distributed encoding system and method.

[0009] Technical solution:

[0010] The present invention first discloses a generative language model distributed coding system, including multiple devices, any two devices can transmit data to each other, and a generative language model is deployed on each device; the generative language model includes a plurality of repeated attention modules, each attention module is divided into three parts, including:

[0011] Part 1, calculation and output of cacheK and cacheV;

[0012] The second part, the storage of cacheK and cacheV;

[0013] In the third part, the matrix multiplication of cacheK and Q of Q / K / V dense is performed, and the result is output to the Softmax operator;

[0014] When storing the second part cacheK and cacheV, cacheK and cacheV are broadcasted at the same time, and the values ​​of cacheK and cacheV are stored in parallel devices.

[0015] As a more optimal implementation, cacheK and cacheV are broadcasted, and the values ​​of cacheK and cacheV are stored in corresponding modules in devices with larger serial numbers than their own.

[0016] The present invention also discloses a generative language model distributed encoding method, which specifically comprises the following steps:

[0017] S1. Device preparation: prepare T devices. Any two devices can transmit data to each other. Each device is numbered sequentially. The generative language model is deployed on each device.

[0018] S2, the user inputs a sentence to be encoded with a length of N;

[0019] S3. The sentence of length N is padded to a length of T*n, where n represents the input length of each device. The sentence of length T*n is evenly divided into T parts and sequentially used as the input of the first part of the attention module Block0 in T devices.

[0020] S4. All devices run the first part of Block0;

[0021] S5. All devices write the output of the first part of their respective Block0 into the second part of their respective Block0 storage;

[0022] S6. All devices broadcast the output of the first part of their respective Block0s; all devices receive the broadcast and store the broadcast content in the second part of Block0;

[0023] S7, run the third part of Block0 and the first part of Block1 simultaneously;

[0024] S8. All devices execute steps S4-S7 for modules Block1 to BlockS;

[0025] S9. All devices take out outputs of length T*n, remove the length padded in step S3, and obtain outputs of length N.

[0026] As a more preferred implementation, in S6, all devices broadcast only to devices with numbers greater than their own; all devices need to wait to receive the broadcast from devices with numbers less than their own, and store the broadcast content in the second part of Block0.

[0027] Preferably, the number of devices T is prepared according to the length N of the sentence and the input length n of the device. By flexibly changing the number of devices T for reasoning, the padding length in S3 is reduced.

[0028] Specifically, the number of devices

[0029] As a more optimal implementation method, for a sentence to be encoded of any length, by decomposing it into N+N+…N+…+M, S1-S9 is executed in a sequential loop to achieve encoding of the sentence to be encoded of any length.

[0030] Preferably, in S6, all devices broadcast the output of the first part of their respective Block0; all devices need to wait to receive the broadcast from a device with a smaller device number than their own, and store the broadcast content in the second part of Block0.

[0031] Beneficial effects of the present invention

[0032] The present invention allows input to be distributed as evenly as possible when a large language model is encoded on multiple devices.

[0033] The present invention allows a large language model to be encoded on multiple devices, with minimal data broadcast during the inference process and minimal waiting time between devices after the input is evenly distributed.

[0034] The present invention allows a large language model to be encoded on N devices, and after the input is evenly distributed, the execution time on each device is close to one-Nth of a single device, that is, the acceleration ratio is close to N. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the attention mechanism structure

[0036] Figure 2 Schematic diagram of parallel acceleration process

[0037] Figure 3 Generate schematic for additional overhead

[0038] Figure 4 Schematic diagram of Ring Self-Attention

[0039] Figure 5 A simplified diagram of a block in the large language model Llama 2

[0040] Figure 6 for Figure 1 Simplified diagram of the attention mechanism module introduced

[0041] Figure 7 A simplified diagram of the large language model structure

[0042] Figure 8 for Figure 7 The model shown is replicated T times and deployed on T devices respectively.

[0043] Fig. 9 for Figure 8 Schematic diagram of data flow of each device shown

[0044] Fig.10 Flowchart for encoding the input using distributed coding to get the first output word "you" DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto:

[0046] A distributed encoding system for generative language models, such as Figure 8 As shown, multiple devices are included, and any two devices can transmit data to each other. Fig. 9 As shown in Figure 2, a generative language model is deployed on each device; Figure 7 As shown in Figure 2, the generative language model includes several repeated attention modules, each of which is divided into three parts: Figure 6 As shown, including:

[0047] Part 1, calculation and output of cacheK and cacheV;

[0048] The second part, the storage of cacheK and cacheV;

[0049] The third part is the matrix multiplication of cacheK and Q of Q / K / V dense, and outputs it to the Softmax operator. Specifically, Q / K / V dense represents the matrix multiplication of Query (Q for short), Key (K for short) and Value (V for short) in the attention mechanism module. Its output is divided into three parts according to Q, K and V, where K is output to a storage cachek, V is output to cachev, and Q is output to the Batch MM0 operator. Batch MM0 is a 3D matrix multiplication, which reads the values ​​of Q of storage cache k and Q / K / V dense for matrix multiplication and outputs it to the Softmax operator.

[0050] When storing the second part cacheK and cacheV, cacheK and cacheV are broadcasted at the same time, and the values ​​of cacheK and cacheV are stored in parallel devices.

[0051] Combination Figure 8 and Fig. 9 , a generative language model distributed encoding method specifically comprises the following steps:

[0052] S1. Device preparation: prepare T devices. Any two devices can transmit data to each other. Each device is numbered sequentially. The generative language model is deployed on each device.

[0053] S2, the user inputs a sentence to be encoded with a length of N;

[0054] S3. The sentence of length N is padded to a length of T*n, where n represents the input length of each device. The sentence of length T*n is evenly divided into T parts and sequentially used as the input of the first part of the attention module Block0 in T devices.

[0055] S4. All devices run the first part of Block0;

[0056] S5. All devices write the output of the first part of their respective Block0 into the second part of their respective Block0 storage;

[0057] S6. All devices broadcast the output of the first part of their respective Block0; all devices receive the broadcast and store the broadcast content in the second part of Block0;

[0058] S7. Run the third part of Block0 and the first part of Block1 simultaneously;

[0059] It should be noted that since all devices need to broadcast their own outputs, assuming that the time required for one device to broadcast to all devices is t, then the time for T devices to broadcast to all devices is T * t. Then S7 needs to wait for T * t time after S5 to start. And if device x only waits for the broadcast of devices with device numbers < x, and assuming that the devices broadcast in ascending order of device numbers, then the waiting time of device x is (x - 1) * t, that is, the waiting time of the first device is 0, the waiting time of the second device is t, the third device is 2 * t, and so on.

[0060] Then when the device executes the broadcast of Block1, at this time, due to the waiting of Block0, the start time of device x for broadcasting is (x - 1) * t later than that of device 0 for broadcasting, and this time is sufficient for the devices with device numbers < x to complete the broadcast of Block1. Therefore, device x does not need to wait any more in Block1 and all subsequent Blocks.

[0061] So when the final model completes the inference, the maximum increase in time introduced by the broadcast is (T - 1) * t, where T is the total number of devices and t is the time required for one device to broadcast in one Block.

[0062] S8. All devices run steps S4 - S7 for modules Block1 to module BlockS;

[0063] S9. All devices take out the output with a length of T * n, remove the length filled in step S3, and obtain the output with a length of N. Specific embodiment:

[0065] Use 1 CPU as the program execution carrier for user input and output interaction. Use two GPU devices to run the inference of the large language model. Select a large language model with two blocks and a maximum input length of 8, and deploy one copy on each GPU device. Assume the user input is "Hello, please introduce yourself."

[0066] As Fig.10 shown is the flowchart for encoding the input using distributed encoding to obtain the first output word "you".

[0067] Mark 1: User input: "Hello, please introduce yourself".

[0068] Mark 2: The user’s input is converted into a token sequence {t0, t1, t2, t3, …, t10} of 11 input length for the model after passing through the tokenizer.

[0069] Tag 3: Pad the input to length 16, {tp,tp,tp,tp,tp,t0,t1,t2,t3,…,t10}.

[0070] Mark 4: Split the padded input into two equal-length parts {tp, tp, tp, tp, tp, t0, t1, t2}, {t3, t4, t5, t6, t7, t8, t9, t10} and send them to GPU0 and GPU1 respectively.

[0071] Mark 5-8: GPU0 and GPU1 start executing the first part of Block0 at the same time, calculating the key and value corresponding to their respective inputs.

[0072] Mark 9: GPU0 stores the key and value, broadcasts them to GPU1, and continues execution.

[0073] Mark 15: GPU0 reads the key and value, Mark 17: calculates the output of Block0.

[0074] Mark 19: GPU0 executes the first part of Block1 and calculates the key and value corresponding to the output of Block0.

[0075] Mark 21: GPU0 stores the key and value and then broadcasts them to GPU1.

[0076] Mark 10: After GPU1 stores the key and value, it broadcasts them to GPU0. Mark 12: and waits for the broadcast data from GPU0 to arrive.

[0077] Mark 14: After receiving the data of Block0 of GPU0, GPU1 stores the key and value of Block0 of GPU0 together with the key and value of Block0 of GPU1.

[0078] Mark 16: GPU1 reads the key and value of GPU0 and GPU1, Mark 18: calculates the output of Block0.

[0079] Mark 20: GPU1 executes the first part of Block1 and calculates the key and value corresponding to the output of Block0.

[0080] Mark 22: After GPU1 stores the key and value, it broadcasts them to GPU0. Mark 24: and waits for the broadcast data from GPU0 to arrive.

[0081] Mark 23 and 25: GPU0 receives and stores the key and value of Block1 of GPU1.

[0082] Mark 26: After receiving the data of Block 1 of GPU 0, GPU 1 stores the key and value of Block 1 of GPU 0 together with the key and value of GPU 1, and continues to calculate the output of Block 1.

[0083] Mark 28: GPU1 reads the key and value of GPU0 and GPU1 and calculates the output of Block1.

[0084] Mark 29: After the CPU waits for GPU1 to complete the execution of Block1, it reads the output of Block1 from GPU1.

[0085] Mark 30-31: The CPU converts the output into tokens and translates it into the first text of the output.

[0086] In this embodiment, the time GPU0 spends waiting for transmission is: 0. The time GPU1 spends waiting for transmission is: the key transmission time of Block0 of GPU0. Therefore, the time added to the model due to parallelism is the key transmission time of Block0 of GPU0. The other computing time of GPU0 and GPU1 is the computing time of 8 tokens. The total time is the computing time of 8 tokens + the key transmission time of GPU0Block0. If tensor parallel is used in this embodiment, and it is assumed that the number of operators that can be used for tensor parallel in each Block is at least 4 (this is the number of operators that can be used for tensorparallel in most large language models, such as Figure 6 The time GPU0 and GPU1 spend waiting for transmission is: 2 transmission times for each operator, that is, 2*4*2=16 transmission times (usually the time length of each transmission of tensor parallel is longer than the time of one transmission of key or value), and the other computing time of GPU0 and GPU1 is the computing time of 16 tokens / 2, which is approximately equal to the computing time of 8 tokens in the execution time of the present invention, and the total time is the computing time of 16 tokens / 2+16 transmission times. Therefore, the present invention greatly reduces the waiting caused by transmission in terms of transmission amount and number of transmissions, and at the same time, distributes other computing time evenly like tensor parallel.

[0087] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A generative language model distributed encoding system, comprising a plurality of devices, any two devices can transmit data to each other, and a generative language model is deployed on each device; characterized in that The generative language model consists of several repeated attention modules, each of which is divided into three parts: Part 1, calculation and output of cacheK and cacheV; The second part, the storage of cacheK and cacheV; In the third part, the matrix multiplication of cacheK and Q / K / V dense is performed and the output is given to the Softmax operator; When storing the second part cacheK and cacheV, cacheK and cacheV are broadcasted at the same time, and the values ​​of cacheK and cacheV are stored in parallel devices.

2. The system according to claim 1, characterized in that Broadcast cacheK and cacheV, and store the values ​​of cacheK and cacheV in the corresponding modules of the device with a larger serial number than itself.

3. A distributed encoding method for a generative language model, characterized in that The specific steps of the encoding method include: S1. Equipment preparation: prepare T devices. Any two devices can transmit data to each other. Each device is numbered sequentially. A generative language model is deployed on each device. S2, the user inputs a sentence to be encoded with a length of N; S3. The sentence of length N is padded to a length of T*n, where n represents the input length of each device. The sentence of length T*n is evenly divided into T parts and sequentially used as the input of the first part of the attention module Block0 in T devices. S4. All devices run the first part of Block0; S5. All devices write the output of the first part of their respective Block0 into the second part of their respective Block0 storage; S6. All devices broadcast the output of the first part of their respective Block0s; all devices receive the broadcast and store the broadcast content in the second part of Block0; S7, run the third part of Block0 and the first part of Block1 simultaneously; S8. All devices execute steps S4-S7 for modules Block1 to BlockS; S9. All devices take out outputs of length T*n, remove the length padded in step S3, and obtain outputs of length N.

4. The method according to claim 3, characterized in that In S6, all devices broadcast only to devices with numbers greater than their own; all devices need to wait to receive the broadcast from devices with numbers less than their own, and store the broadcast content in the second part of Block0.

5. The method according to claim 3, characterized in that Prepare the number of devices T according to the length N of the sentence and the input length n of the device.

6. The method according to claim 5, characterized in that Number of devices 7. The method according to claim 3, characterized in that For a sentence to be encoded of any length, by decomposing it into N+N+…N+…+M, S1-S9 are executed in a loop in sequence to achieve encoding of the sentence to be encoded of any length.

8. The method according to claim 7, characterized in that In S6, all devices broadcast the output of the first part of their respective Block0s; all devices need to wait to receive the broadcast from a device with a smaller device number than their own, and store the broadcast content in the second part of Block0.

Images