Calculation method of self-attention operator, operator calculation equipment, system and chip

Abstract

The embodiment of the invention discloses a calculation method of a self-attention operator, operator calculation equipment, a system and a chip, which are used for unloading the self-attention operator from reasoning equipment to the operator calculation equipment, reducing the data transmission quantity, reducing the requirement on the SSD transmission bandwidth and improving the reasoning real-time performance at the same time. The self-attention operator computing system comprises: a reasoning device for calculating a first increment KVcache of a first decoder in a plurality of decoders according to a first output element token, and sending the first increment KVcache to an operator computing device; the operator calculation equipment is used for executing calculation of a first self-attention operator of the first decoder according to the total amount KVcache and the first increment KVcache of the first decoder to obtain a first calculation result and sending the first calculation result to the reasoning equipment, and the reasoning equipment is used for receiving the first calculation result and sending the first calculation result to the second decoder according to the total amount KVcache of the first decoder and the first increment KVcache of the first decoder. The total amount KVcache of the first decoder is stored in the operator computing equipment in advance.

Description

Technical Field

[0001] This application relates to the fields of computer technology and storage, and in particular to a calculation method for self-attention operators, operator calculation devices, systems, and chips. Background Art

[0002] Large language models, also known as large language models or large models (Large Language Model, LLM), aim to understand and generate natural language by learning a large amount of text data to predict content related to a given text. Among them, the third-generation General Pre-trained Transformer-3 (GPT3) is a specific application of large language models, demonstrating the powerful capabilities and possibilities of large language models.

[0003] The architecture of GPT3 is based on the Transformer model, and the Transformer model is a deep learning model based on the self-attention mechanism. When the Transformer model performs self-attention operator calculations, it calculates key-value (KV) vectors based on the vectors corresponding to each element in the currently received input sequence. The calculated K vectors are stored in a matrix, usually referred to as Kcache (cache), and the V vectors are stored in a matrix, usually referred to as Vcache. Collectively, they are called KVcache. Generally, as the sequence lengths of questions and answers increase, the data volume in Kcache and Vcache during the running process will increase sharply. Then, the memory occupied by Kcache and Vcache gradually increases, resulting in a large consumption of High Bandwidth Memory (HBM) on the Graphic Process Unit (GPU), challenging the capacity of HBM.

[0004] In the prior art, to solve the above technical problems, KVcache is stored on a solid-state drive, but in this case, there are also high requirements for the transmission bandwidth between the GPU side and the solid-state drive. Summary of the Invention

[0005] The embodiments of this application provide a calculation method for self-attention operators, operator calculation devices, systems, and chips, which can reduce the amount of data transmitted between the GPU and the SSD, can reduce the requirements for the SSD transmission bandwidth, make the SSD transmission bandwidth match the computing power on the GPU side, and improve the real-time performance of inference.

[0006] In a first aspect, an embodiment of the present application provides a computing system, including an inference device and an operator computing device. Taking the inference device as a GPU and the operator computing device as an SSD as an example, a transformer model runs in the GPU, and the transformer model includes multiple decoders. The GPU calculates a first incremental KV cache of a first decoder according to a first output element token, and sends the first incremental KV cache to the SSD, where the first output token is output by the GPU when performing the first token inference on the received input sequence; then the SSD merges the full KV cache of the first decoder and the first incremental KV cache, and performs the first self-attention operator calculation of the first decoder according to the merged KV cache to obtain a first calculation result, where the full KV cache of the first decoder is calculated by the GPU according to the received input sequence and pre-stored in the SSD. In the embodiment of the present application, the self-attention operator calculation of each decoder is performed on the SSD, that is, the calculation of the self-attention operator is offloaded from the GPU to the SSD. Since the full KV cache has been stored on the SSD, only the incremental KV cache needs to be synchronized between the GPU and the SSD, and the data volume of the incremental KV cache is much smaller than that of the full KV cache, so the data transmission volume can be reduced, the transmission bandwidth requirement for the SSD is reduced, the transmission bandwidth of the SSD can match the computing power on the GPU side, and the real-time performance of inference is improved.

[0007] Wherein, in the embodiment of the present application, after the GPU calculates the full KV cache (the full KV cache includes the full KV cache of each decoder in multiple decoders), the first output token can be inferred according to the output of the last decoder. And the first output token is used as the input for the next inference. Since the full KV cache of the input sequence has been calculated and stored in the SSD, in this inference, only the first incremental KV cache corresponding to the first output token needs to be calculated, thereby reducing the calculation amount and improving the real-time performance of inference.

[0008] In a possible implementation manner, after the SSD performs the first self-attention operator calculation of the first decoder to obtain a first calculation result, the first calculation result is sent to the GPU. The GPU calculates the first output of the first decoder, and calculates a second incremental KV cache of a second decoder according to the first output. The SSD performs the second self-attention operator calculation of the second decoder to obtain a second calculation result. Of course, in the specific implementation process, it is necessary to iterate in sequence and go through the processing of more decoders to improve the accuracy of inference.

[0009] In a possible implementation, the computing system in the embodiments of the present application further includes other inference devices and other operator computing devices, that is, other GPUs and other SSDs are also included. In this way, when calculating the incremental KVcache, it can be synchronously performed by the GPU and other GPUs. Similarly, the calculation of the self-attention operator can also be synchronously performed by the SSD and other SSDs. Since multiple GPUs and multiple SSDs perform parallel computing, the computing efficiency can be improved, the computing time can be shortened, so as to improve the real-time performance of inference. At the same time, since multiple SSDs bear the transmission bandwidth simultaneously, the requirement for the transmission bandwidth between the SSD and the GPU can be further reduced.

[0010] In a possible implementation, the inference device includes a graphics processing unit GPU or a general-purpose graphics processing unit GPGPU.

[0011] In a possible implementation, the operator computing device includes a solid-state drive SSD, a data processor DPU, a neural network processor NPU, or other devices capable of calculating the self-attention operator.

[0012] In a possible implementation, the element token can be a token, and the token can be a Chinese character, a word, or a punctuation mark. The token can also be an image block.

[0013] In a second aspect, the embodiments of the present application further provide a method for calculating a self-attention operator, which is executed by an operator computing device. The functions or technical effects achieved during the execution of this calculation method by the operator computing device can refer to the functions or technical effects brought by the first aspect and any possible implementation manner in the first aspect.

[0014] In a third aspect, the present application further provides an operator computing device, including: an interface for communicating with an inference device; a controller for executing the method described in the second aspect and any possible implementation manner in the second aspect. The functions or technical effects achieved during the execution of the calculation method by the controller can refer to the functions or technical effects brought by the first aspect and any possible implementation manner in the first aspect. It should be noted that when the operator computing device is an SSD, the controller includes an SSD controller. In addition to the SSD controller, other processors can also be included for calculating the self-attention operator; while when the operator computing device is a DPU or an NPU, the controller is the processor for executing the method described in the second aspect and any possible implementation manner in the second aspect.

[0015] Fourthly, the present application further provides a chip, including: a power supply circuit for power supply; a processor for obtaining a first incremental KV cache of a first decoder and a full KV cache of the first decoder among multiple decoders; and then calculating a first self-attention operator of the first decoder according to the full KV cache and the first incremental KV cache of the first decoder, and obtaining and outputting a first calculation result. The functions or technical effects achieved by the processor in the process of executing the above steps can refer to the functions or technical effects brought by the implementation manners in the first aspect above.

[0016] Fifthly, the present application further provides a computer program product including instructions, which when running on a computer, causes the computer to execute the following steps: obtaining a first incremental KV cache of a first decoder and a full KV cache of the first decoder among multiple decoders; and then calculating a first self-attention operator of the first decoder according to the full KV cache and the first incremental KV cache of the first decoder, and obtaining and outputting a first calculation result. The functions or technical effects achieved by the computer in the process of executing the above steps can refer to the functions or technical effects brought by the implementation manners in the first aspect above. Description of the Drawings

[0017] Figure 1 is a schematic architecture diagram of a transformer model in the prior art;

[0018] Figure 2 is a schematic architecture diagram of a computing system in the prior art;

[0019] Figure 3 is a schematic flowchart of self-attention operator calculation in the prior art;

[0020] Figure 4 is a schematic architecture diagram of a computing system provided by an embodiment of the present application;

[0021] Figure 5 is a schematic architecture diagram of another computing system provided by an embodiment of the present application;

[0022] Figure 6 is a schematic architecture diagram of an SSD provided by an embodiment of the present application;

[0023] Figure 7 is a schematic flowchart of self-attention operator calculation provided by an embodiment of the present application. Detailed Embodiments

[0024] Before describing the calculation method of the self-attention operator provided by the present application, key technical terms involved in the present application are first explained.

[0025] 1. Transformer Model

[0026] The Transformer model is a transduction model that relies entirely on self-attention to compute its input and output representations, and it abandons traditional Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The Transformer model architecture typically includes three types. As an example, the Encoder-Decoder structure or only the Encoder Only or only the Decoder Only. Among them, only the Decoder Only is the currently more commonly used structure.

[0027] Please refer to Figure 1 , when the Transformer model only includes the decoder, the number of decoders can be multiple. The structures of each decoder in the multiple decoders are the same. Each decoder can include two sub-layers, namely the self-attention layer and the feed-forward neural network layer. The output of the last decoder, after being processed by softmax, can obtain the output element token of this round of inference.

[0028] Softmax is a mathematical function that is usually used to transform a set of arbitrary real numbers into real numbers representing a probability distribution. In essence, it is a normalization function that can transform a set of arbitrary real values into probability values between [0, 1].

[0029] In Natural Language Processing (NLP), a token refers to a basic unit in text, which can usually be a word, a phrase, a punctuation mark, etc.; in Computer Vision (CV), a token refers to cutting an image into non-overlapping image patches. Therefore, taking NLP as an example, if the input sequence S is "The sky is overcast", then the input element tokens are "The sky", "is", "overcast". The output sequence corresponding to the input sequence is "The pagoda subdues the river demon", then the output tokens are "The pagoda", "subdues", "river demon".

[0030] The input sequence S enters Figure 1Before the decoder 1 shown in the figure, each input token is first converted into a vector by the word embedding method. Among them, embedding is to represent an object with a low-dimensional dense vector. Here, the object can be a character or a word. The embedding vector can express certain features of the object, and the distance between two vectors reflects the similarity between the objects. The input of the decoder 1 is the vector after the word embedding method conversion. Continuing with the above example, the input tokens are "Sky King", "Cover", and "Earth Tiger". First, each token is converted into a vector based on the word embedding algorithm, expressed as X = [x1 x2 x3], where x1, x2, and x3 represent the vectors corresponding to the 3 tokens respectively. Then X is the input of the decoder 1. In other decoders, the input of each decoder is the output of the previous layer decoder. In other words, the output of each decoder is the input of the next layer decoder. Similarly, the output of the decoder L, after being processed by softmax, can infer the output tokens corresponding to the input sequence.

[0031] 2. Self-attention mechanism

[0032] The self-attention mechanism calculates the degree of association between each token in the input sequence and other tokens, thereby assigning different weights to each token. This weight represents the importance between different tokens, enabling the transformer model to better capture the association information in the sequence. Continuing with the above example, after obtaining X = [x1 x2 x(3)], it is multiplied by the weight matrices W q 、W k 、W v respectively. That is, X * W q = Q, X * W k = K, X * W v = V, and the matrices K, Q, and V can be obtained. Q is used to represent the attention weights of each token, K is used to represent the weight index, and V is used to represent the token vector. These three matrices are used for different tasks. The first two matrices are multiplied, that is, Q is multiplied by the transpose of K, and then normalized by softmax can represent the importance of each token relative to other tokens. Multiplying by V gives the calculation result of self-attention.

[0033] 3. KV cache

[0034] When the Transformer model performs the self-attention operator calculation, it calculates the key-value (KV) vectors based on the vectors corresponding to each element in the currently received input sequence. The calculated K vectors are stored in a matrix, which is usually called the K cache (cache), and the calculated V vectors are stored in a matrix, which is usually called the V cache. Together, they are called the KV cache.

[0035] The inference process of the Transformer model can be divided into two stages:

[0036] (1) Full processing stage: The full processing stage can also be called the prefill stage, which occurs during the calculation of the first output token. At this time, the cache is empty. During the calculation, it is necessary to calculate and save the K cache and V cache for each decoder. The cache is filled when the token is output.

[0037] (2) Answer generation stage: It can also be called the decoding stage, or the stage of using the K cache and V cache. It occurs during the calculation of the second output token to the last output token. At this time, the cache has values. In each round of inference, only the cache needs to be read, and at the same time, the newly calculated Key and Value in the current round are appended and written to the cache.

[0038] Continuing with the above example, if the input sequence is "The sky is overcast, and the earth is fierce", and the output sequence is "The pagoda subdues the river demon", then during the inference of the first output token "pagoda", the K and V corresponding to each decoder are called the KV cache, and the calculated KV cache is stored in the SSD. When inferring the second output token "subdues", using the first output token "pagoda" as the input, it is only necessary to calculate the incremental k and incremental v corresponding to "pagoda", without having to recalculate "The sky is overcast, and the earth is fierce", so as to reduce the computational complexity.

[0039] The following will be described in conjunction with the accompanying drawings how the self-attention operator calculation is performed in the prior art. Please refer to Figure 2, which is a schematic diagram of a system architecture in the prior art. The system architecture diagram includes a GPU 201 (Graphic Process Unit, GPU) and a solid-state drive 202 (Solid State Disk or Solid State Drive, SSD). The GPU 201 is located in an Artificial Intelligence (AI) server. A transformer model runs in the GPU 201, and the transformer model includes L decoders. Here, the SSD 202 is mainly used to store the full KV cache and incremental KV cache calculated by the GPU 201 during operation.

[0040] Please refer to Figure 3 , in the prefill stage:

[0041] The GPU 201 generates a full KV cache and sends the full KV cache to the SSD 202. The full KV cache includes the full KV cache corresponding to each decoder. This step is represented by serial number ①;

[0042] Based on the output of the last decoder, the GPU 201 determines the first output token, that is, the token output for the first time. This step is represented by serial number ②.

[0043] decoding stage:

[0044] First decoder: The GPU 201 calculates the first incremental KV cache of the first decoder according to the first output token. This step is represented by serial number ③; and the GPU 201 reads the full KV cache of the first decoder from the SSD 202, and combines the full KV cache of the first decoder ( Figure 3 represented by full KV cache 1 herein) and the first incremental KV cache. This step is represented by serial number ④, and then performs the calculation of the self-attention operator. This step is represented by serial number ⑤, and after subsequent Feedforward Neural Network (FFN) processing, the first output result of the first decoder is obtained. At the same time, the GPU 201 sends the first incremental KV cache to the SSD 202; the SSD 202 combines and stores the first incremental KV cache with the full KV cache of the first decoder. This step is represented by serial number ⑥;

[0045] Second decoder: The GPU 201 uses the first output result as the input of the second decoder to calculate the second incremental KV cache of the second decoder. The GPU 201 reads the full KV cache of the second decoder from the SSD 202, and the full KV cache of the second decoder (Figure 3 After the first full KV cache (represented by the full KV cache 2 in Chinese) is merged with the second incremental KV cache, the self-attention operator is calculated and then the subsequent FFN network processing is performed to obtain the second output result of the second decoder. At the same time, GPU 201 sends the second incremental KV cache to SSD 202; SSD 202 merges and stores the second incremental KV cache with the full KV cache of the second decoder;

[0046] And so on;

[0047] The L-th decoder: GPU 201 uses the (L - 1)-th output result as the input of the L-th decoder, calculates the L-th incremental KV cache. GPU 201 reads the full KV cache of the L-th decoder from SSD 202, and after merging the full KV cache of the L-th decoder ( Figure 3 represented by the full KV cache L in Chinese) and the L-th incremental KV cache, performs the calculation of the self-attention operator and the subsequent FFN network processing to obtain the L-th output result of the L-th decoder. GPU 201 sends the L-th incremental KV cache to SSD 202, and SSD 202 merges and stores the L-th incremental KV cache with the full KV cache of the L-th decoder. After GPU 201 performs subsequent processing on the output result of the L-th decoder, the second output token can be inferred, that is, the token output for the second time.

[0048] And so on;

[0049] Until the last output token is inferred, that is, the token output for the third time.

[0050] As can be seen from the above inference process, during the process of completing one token inference, SSD 202 needs to synchronize the full KV cache L times to GPU 201. Taking the GPT3 175B model as an example for the data volume of the full KV cache, the number of hidden layers of this model is 12288 layers, the number of decoders is 96, and Kcache and Vcache are saved in float16. The input sequence length and the output sequence length are both 2048. Then the approximate data volume of the first calculation of Kcache and Vcache is s * h * l * 2 * 2, where s is the input sequence length, the first 2 represents K and V, and the second 2 represents float16. Then the result is 9 gigabytes (GB). When outputting the last element token, the data volume of Kcache and Vcache is approximately s * h * l * 2 * 2 = 18GB, where s is 4096. It can be seen that the data volume of the full KV cache is relatively large, which requires a relatively high transmission bandwidth between SSD 202 and GPU 201.

[0051] To this end, the present application proposes a computing system that offloads the calculation of the self-attention operator to a solid-state drive or other device. Since the calculation of the self-attention operator in the prior art requires multiple synchronizations of the full KVcache between the GPU and the SSD, and it can be seen from the above example that the data volume of the full KVcache is large, the bandwidth requirement between the GPU and the SSD is also high, which makes the bandwidth and computing power unmatched, thereby affecting the real-time performance of reasoning. The present application offloads the calculation of the self-attention operator to the SSD, so that the incremental KVcache can be synchronized between the GPU and the SSD. Since the data volume of the incremental KVcache is much smaller than the full KVcache, the data transmission volume is reduced, the requirements for the SSD transmission bandwidth are reduced, and the real-time performance of reasoning is improved.

[0052] In order to better understand the technical solutions of the embodiments of the present application, the following briefly introduces the possible application scenarios of the embodiments of the present application.

[0053] Scenario 1: Natural Language Processing Scenario

[0054] A computing system provided in the present application can be applied to a natural language processing scenario, which includes a user device, an inference device, and an operator computing device. The user device may include a user and an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of natural language data processing. As the initiator of language questions and answers or queries and other requests, the user usually initiates the request through the user device. The inference device can be a GPU or a general-purpose graphics processing unit (GPGPU) with data processing capabilities. The GPU or GPGPU is located in the AI server. The inference device receives an input sequence from the intelligent terminal through an interactive interface, processes the input sequence, and obtains a corresponding output. The operator computing device can be a storage device with operator computing capabilities, such as an SSD, or other devices with operator computing capabilities, such as a data processor (DPU) or a neural network processor (NPU). The operator computing device can also be a chip, which can be an SSD controller, or other processors in the SSD controller other than its own processor. As an example, the other processor can be a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0055] As an example, if the input sequence is a sequence of words in the original language, such as a sentence or a phrase, the output sequence can be a summary of the input sequence in the original language, that is, a sequence with fewer words than the input sequence but retaining the basic meaning of the input sequence.

[0056] As another example, if the input sequence of the intelligent terminal is a sequence of words forming a question, the output sequence can be a sequence of words forming an answer to the question.

[0057] As another example, if the input sequence of the intelligent terminal is a sequence of audio data representing the spoken words, the output sequence can be a sequence of glyphs, characters, or words representing the spoken words, that is, a transcription of the input sequence.

[0058] As another example, if the input sequence is a sequence of words in the original language, such as a sentence or a phrase, the output sequence can be a translation of the input sequence into the target language, that is, a sequence of words in the target language that represents the sequence of words in the original language.

[0059] It should also be noted here that in the specific implementation process, the user device can be the inference device described above. It directly receives the input from the user and is directly processed by the hardware of the intelligent terminal itself.

[0060] Scenario 2: Computer vision field

[0061] A computing system provided by this application can be applied to the computer vision field. Specifically, it can be a part of an image processing system. In this scenario, it includes a user device, an inference device, and an operator computing device. Among them, the user device, the inference device, and the operator computing device can refer to the examples in Scenario 1 and will not be elaborated here.

[0062] As an example, the input sequence can be an image, that is, a sequence of color values from the image, and the output sequence can be a sequence of text describing the image. As another example, the input sequence can be a sequence of text or different contexts, and the output sequence can be an image describing the context.

[0063] The following introduces the schematic diagram of the system framework adopted in the embodiments of this application.

[0064] The system architecture provided by the embodiments of this application includes an inference device and an operator computing device. Among them, the inference device can be a GPU or a GPGPU. The GPU or GPGPU is located in the AI server, and the number of inference devices can be one or more.

[0065] The operator computing device can be an SSD or other devices capable of computing the self-attention operator, such as a DPU or an NPU. The number of operator computing devices can be one or multiple. The SSD here is different from the SSD in Figure 2 in that the SSD here is not only used to store the full KV cache and incremental KV cache output during GPU computing, but also undertakes the calculation of the self-attention operator of each decoder.

[0066] As an example, please refer to Figure 4 , the system architecture includes 8 GPUs 401 and one SSD 402. In this example, the 8 GPUs 401 are communicatively connected to each other, and all 8 GPUs 401 are communicatively connected to the SSD 402. That is to say, some of the processed data of the 8 GPUs 401 need to be sent to the SSD 402, and some of the processed data can be the full KV cache calculated by each GPU 401 and the incremental KV cache corresponding to each decoder during one inference process. After receiving the full KV cache and incremental KV cache sent by each GPU 401, the SSD 402 performs the calculation of the corresponding self-attention operator according to the full KV cache and incremental KV cache sent by each GPU, and then sends the calculated result to the corresponding GPU 401.

[0067] As another example, please refer to Figure 5 , the system architecture includes 8 GPUs 501 and multiple SSDs 502. Among them, the number of SSDs 502 can match the number of GPUs 501, such as 8 SSDs 501. Of course, the number of SSDs 502 can also not match the number of GPUs 501. For example, the number of GPUs 501 is 8 and the number of SSDs 502 is 4. In this case, one SSD 502 processes the data sent by 2 GPUs 501. Whether the number of GPUs 501 matches the number of SSDs 502 or not, when the networking of GPUs 501 and SSDs 502 is completed, the communication objects of GPUs 501 and SSDs 502 are preset. In other words, the GPUs 501 can know which SSDs 502 to send data to, and the corresponding SSDs 502 can also know which GPUs 501 to send data to. In this example, the communication methods between GPUs 501 and SSDs 502 include but are not limited to the following two:

[0068] Method 1: There is a corresponding communication connection relationship between the GPU 501 and the SSD 502, and one GPU 501 is communicatively connected to one SSD 502.

[0069] Taking the example where the number of GPUs 501 is 8 and the number of SSDs 502 is 8, in this method, there is a full interconnection between each GPU 501 and each SSD 502. If the 8 GPUs 501 are respectively represented as GPU1, GPU2, GPU3, …… GPU8, and the 8 SSDs are respectively represented as SSD1, SSD2, SSD3, …… SSD8, then, GPU1 is communicatively connected to SSD1, GPU2 is communicatively connected to SSD2, GPU3 is communicatively connected to SSD3, …… GPU8 is communicatively connected to SSD8, and the 8 GPUs are communicatively connected to each other, and the 8 SSDs are also communicatively connected to each other.

[0070] Continuing with the above example, when GPU1 is communicatively connected to SSD1, GPU1 can communicate with SSD1 through the CPU in the AI server, or GPU1 can also perform direct data transfer with SSD1, bypassing the CPU in the AI server, so as to reduce the data transfer latency and CPU overhead. In this way, the communication protocol between GPU 501 and SSD 502 can be based on the Peripheral Component Interconnect Express (PCIe), or other communication protocols.

[0071] Similarly, the communication between GPUs also includes two methods. As an example, to send the data in GPU1 to GPU2, it is necessary to copy the data to be sent in GPU1 to the fixed shared memory on the server through the CPU and PCIe in the AI server. Then, the data to be sent is copied from the shared memory to GPU2 through the CPU and PCIe bus. As another example, GPU1 can also perform direct data transfer with GPU2, that is, data communication between GPU1 and GPU2 is carried out through GPUDierect P2P (Peer to peer), where GPUDierect P2P is an optimized GPU communication technology that allows direct data transfer between different GPUs, avoiding the additional overhead of passing through the inference device memory.

[0072] In Method 2, there is a full interconnection between each GPU 501 and each SSD 502.

[0073] Each of the eight GPUs 501 is communicatively connected to one of the eight SSDs 502. Continuing with the example in Method 1, GPUs 1 - 8 are all communicatively connected to SSD 1. The eight GPUs are communicatively connected to each other, and the eight SSDs are also communicatively connected to each other. In this way, after SSD 1 receives the data sent by GPUs 1 - 8, it distributes the received data to SSDs 2 - 8. Among them, the communication methods between GPU 501 and SSD 502 and between GPU 501 and GPU 501 can refer to the description in Method 1 and will not be elaborated here.

[0074] For the system architecture in the above - mentioned another example, since it includes multiple SSDs, one SSD processes the data sent by one GPU. For example, SSD 1 receives the full - volume KVcache and incremental KVcache sent by GPU 1, and then calculates the attention operator based on the received data,... SSD 8 receives the full - volume KVcache and incremental KVcache sent by GPU 8, and then calculates the self - attention operator based on the received data. In this way, multiple GPUs and multiple SSDs can perform parallel data processing, improving the computing efficiency and shortening the computing time, thereby improving the inference real - time performance. At the same time, since multiple SSDs jointly bear the bandwidth, the requirement for the transmission bandwidth between the SSD and the GPU can be further reduced.

[0075] After introducing the computing system, the system architecture of SSD 402 in the above Figure 4 will be further introduced. The system architecture of SSD 502 in Figure 5 can refer to the description of the system architecture of SSD 402 and will not be elaborated here.

[0076] Please refer to Figure 6 , this application embodiment discloses an SSD 402, which includes a controller 4021, a flash memory array 4022 composed of multiple flash memory particles, and a cache 4023. Among them, the controller 4021 is respectively connected to the inference device, the flash memory, and the cache.

[0077] The controller 4021 is responsible for some complex tasks, such as managing data storage, maintaining the performance and service life of SSD 402, etc. As an example, the controller 4021 is used to receive the access command sent by the inference device, parse the command and convert it into a command that can directly access the flash memory array and send it to the flash memory array, then obtain the access result and return the result to the inference device. The controller 4021 is usually presented in the form of an application - specific integrated circuit, and can also be implemented based on a field - programmable gate array or based on a central processing unit. In practical applications, considering factors such as cost, performance, and power consumption, the controller is usually made into an ASIC chip.

[0078] The flash memory array 4022 is used to store various data. Specifically, the flash memory array 4022 may include one or more flash memory particles, and each flash memory particle is usually presented in the form of a chip. The specific type of the flash memory is NAND flash memory. It should be noted that although the flash memory is taken as an example here, in fact, it can also be other types of non-volatile memories, such as Phase Change Memory (PCM), Resistive Random Access Memory (RRAM), etc., which does not affect the technical solution of this application.

[0079] The cache 4023 is optional in the SSD 402 and is usually implemented by Dynamic Random Access Memory (DRAM). It is used to store various data generated during operation, which is beneficial to improving the response speed of the controller to the commands of the inference device.

[0080] The controller 4021 also includes an inference device interface and several channel controllers. Among them, the inference device interface is used to communicate with the inference device, and the inference device here can refer to any device such as a server, a personal computer, or an array controller. Through several channel controllers, the SSD controller 4021 can operate multiple flash memory chips in parallel, thereby improving the underlying bandwidth.

[0081] The controller 4021 usually includes a processor, whose function is like a command center and sends all operation requests of the SSD 402. For example, the processor executes functions such as reading / writing data, garbage collection, and wear leveling through the firmware in the buffer. In this application, the processor of the SSD controller 501 can receive data sent by the GPU through the interface and calculate the self-attention operator.

[0082] In some embodiments, in addition to the original processor, the controller 4021 further includes an FPGA, an ASIC, or other processing chips for calculating the self-attention operator. In this way, it will not affect the processing operations of the original processor and reduces the burden on the processor.

[0083] In some other embodiments, when the operator calculation device is a DPU or an NPU, the controller 4021 can be the processor itself.

[0084] In a first aspect, in the embodiments of this application, in combination with Figure 4 the shown computing system and Figure 6 the shown SSD 402 architecture, taking the controller 4021 as the SSD controller as an example, the calculation method of the self-attention operator is introduced in detail. For details, please refer to Figure 7, when the controller 4021 includes other processing chips in addition to the SSD controller, the other processing chips perform the self-attention operator calculation, and the execution process can refer to the SSD controller. Similarly, when the operator calculation device is a DPU or an NPU, the DPU or NPU processor performs the self-attention operator calculation, and the execution process can also refer to the SSD controller, which will not be elaborated here.

[0085] Case 1

[0086] Taking the number of GPUs 401 as 8 and the number of SSDs 402 as 1 as an example, since the processing process of the data received by each of the 8 GPUs 401 is the same, only one of the GPUs 401 will be described as an example. Specifically:

[0087] GPU 401 is used to obtain the input sequence; here, the input sequence can be a sentence, a piece of speech, or an image. Among them, the length of the input sequence can be 1 to 16384, or even longer, which is not limited here. In the embodiments of the present application, for the convenience of description, taking the input sequence length as 3 as an example, if the obtained input sequence is "The sky is overcast, the earth is fierce", then the input sequence can include 3 input tokens, namely "The sky", "covers", and "the earth is fierce".

[0088] GPU 401 calculates the full-scale KVcache based on the input sequence, sends the full-scale KVcache to SSD 402, and pre-stores it in SSD 402. Among them, this step is represented by the serial number ①;

[0089] As an example, the input sequence is represented by S, specifically "The sky is overcast, the earth is fierce". For each input token, GPU 401 will convert each input token into a high-dimensional vector through the word embedding method. Taking the word embedding dimension as 12288 as an example, after X undergoes word embedding processing, a matrix with a dimension of 3×12288 will be obtained.

[0090] After converting the input sequence S into the matrix X, using the weight matrices W q , W k and W v , calculate the K1, Q1, and V1 matrices, that is, Q1 = W q *X, K1 = W k *X, V1 = W v *X. Here, K1, Q1, and V1 correspond to the first decoder. That is to say, each decoder included in the transformer model will calculate the corresponding K, Q, and V. K1 and V1 are the full-scale KVcache corresponding to the first decoder. W q , W and W vIf the dimensions of the matrices are all 12288×1536, then the dimensions of K, Q, and V are all 3×1536. It should be noted here that if there is only one GPU 401, the weight matrices W q 、W k and W v would have dimensions of 12288×12288. In this application, to accelerate the calculation and inference process, 8 GPUs 401 are used for parallel computing. Then the dimensions of the weight matrices on each GPU should be 12288×1536, but the weight matrices on each GPU are not the same. The following will be described separately according to the two stages of inference.

[0091] prefill stage:

[0092] (1) After calculating the K1, Q1, and V1 matrices, enter the first decoder. In the first decoder, GPU 401 performs the calculation of the first attention operator of the first decoder according to the calculated K1, Q1, V1, and the calculation formula of the self-attention operator to obtain the first calculation result, and input the first calculation result into the feed-forward neural network layer to obtain the first output result of the first decoder, and send the calculated K1 and V1 to SSD 402. Here, K1 and V1 refer to the full KV cache of the first decoder. The reason it is called full is that the calculated KV cache includes all input tokens in the input sequence;

[0093] (2) Use the first output result as the input of the second decoder. GPU 401 calculates K2, Q2, and V2 of the second decoder according to the first output result, and then performs the calculation of the second self-attention operator of the second decoder according to K2, Q2, and V2 to obtain the second calculation result, and input the second calculation result into the feed-forward neural network layer to obtain the second output of the second decoder, and at the same time send K2 and V2 to SSD 402. Among them, K2 and V2 are the full KV cache of the second decoder;

[0094] And so on;

[0095] (L) Use the (L - 1)th output result as the input of the Lth decoder. GPU 401 calculates K L 、Q L 、V L of the Lth decoder according to the (L - 1)th output result, and then performs the calculation of the Lth self-attention operator of the Lth decoder according to K L 、Q L 、V L to obtain the Lth calculation result, and input the Lth calculation result into the feed-forward neural network layer to obtain the output of the Lth decoder, and at the same time send K L 、VL Sent to the SSD, where K L , V L is the full KV cache of the L-th decoder;

[0096] Up to this point, that is, after the L-th step is completed, the full KV cache can be calculated, including the full KV caches of each decoder in multiple decoders. At the same time, the GPU 401 sends the full KV cache to the SSD 402, and the SSD controller 4021 receives the full KV cache sent by the GPU 401 through the interface. The SSD controller 4021 stores the received full KV cache in the flash memory array 4022, or the SSD controller 402 first stores the received full KV cache in the buffer 4023 and then permanently stores it in the flash memory array 4022.

[0097] After calculating the full KV cache, the GPU 401 is further used for:

[0098] According to the output of the last decoder in the multiple decoders, infer the first output token corresponding to the input sequence; Continuing with the above example, for the first token inference, the inferred first output token is "pagoda", and this step is represented by the serial number ②.

[0099] Correspondingly, the GPU 401 is specifically used for: calculating the first incremental KV cache according to the first output token. The stage corresponding to this step is the decoding stage mentioned above.

[0100] Decoding stage:

[0101] (1) The first decoder

[0102] After determining the first output token, GPU 401 uses the first output token as the input for the next token inference. Continuing with the above example, the input sequence is "The sky covers the ground with a tiger", and the K and V corresponding to the initial input sequence have been stored in SSD 402. To avoid duplicate calculations, only the k, q, and v of the first output token need to be calculated here, and the K and V corresponding to the initial input sequence can be read from SSD 402. Compared with the full KV cache, the k and v calculated based on the newly added output token here can be called the incremental KV cache. As an example, the first output token is transformed into a vector with a dimension of 1×1536 using the word embedding method. Based on the weight matrix, q1, Δk1, and Δv1 can be calculated. Among them, Δk1 and Δv1 are the first incremental KV cache, and this step is represented by serial number ③. At the same time, GPU 401 sends q1, Δk1, and Δv1 to SSD 402, and this step is represented by serial number ④.

[0103] The SSD controller 4021 receives the first incremental KV cache of the first decoder sent by GPU 401 through the interface; then the SSD controller 4021 reads the full KV cache of the first decoder (represented by KVcache1 in Figure 7 ) from the flash memory array 4022. This step is represented by serial number ⑤, combines the full KV cache1 and the first incremental KV cache, performs the calculation of the first self-attention operator of the first decoder, obtains the first calculation result, and sends the first calculation result to GPU 401 through the interface. This step is represented by serial number ⑥.

[0104] In the specific calculation process, the SSD controller 4021 combines the full KV cache1 and the first incremental KV cache vertically. Among them, vertical combination means combination in the row direction of the matrix, and the combination symbol can be represented by [;]. The combined full KV cache is obtained, and the dimension of the combined full KV cache is 4×1536. According to the combined full KV cache, q1, and the formula Perform the calculation of the first self-attention operator of the first decoder to obtain the first calculation result with a dimension of 1×1536. The SSD controller 4021 sends the calculated first calculation result to GPU 401 through the interface; at the same time, stores the combined full KV cache in the flash memory array 4022.

[0105] GPU 401 inputs the first calculation result into the feed-forward neural network layer. After being processed by the feed-forward neural network layer, the first output result of the first decoder is obtained, and the vector dimension of the first output result is also 1×1536; The original text to be translated is as below which wraped by :

[0106] (2) Second Decoder

[0107] Based on the first output result, GPU 401 calculates the second incremental KV cache, obtaining q2, Δk2, and Δv2. Among them, Δk2 and Δv2 are the second incremental KV cache of the second decoder, and q2, Δk2, and Δv2 are sent to SSD 402;

[0108] The SSD controller 4021 receives the second incremental KV cache sent by GPU 401 through the interface. At the same time, the SSD controller 4021 reads the full KV cache of the second decoder from the flash memory array 4022 (represented by KVcache2 in Figure 7 ), and merges the second incremental KV cache and the full KV cache2 vertically to obtain the merged full KV cache. The dimension of the merged full KV cache is 4×1536. According to q2, the merged full KV cache, and the formula Perform the calculation of the second self-attention operator of the second decoder to obtain the second calculation result, and send the second calculation result to GPU 401 through the interface;

[0109] GPU 401 inputs the second calculation result into the feed-forward neural network layer of the second decoder. After being processed by the feed-forward neural network layer of the second decoder, the second output result of the second decoder can be obtained, and the second output result is used as the input of the next decoder;

[0110] And so on;

[0111] (L) The L-th Decoder

[0112] Based on the (L - 1)-th output result, GPU 401 calculates the L-th incremental KV cache, obtaining q L , Δk L , Δv L . Among them, Δk L , Δv L are the L-th incremental KV cache of the L-th decoder, and q L , Δk L , Δv L are sent to SSD 402;

[0113] The SSD controller 4021 receives the L-th incremental KV cache sent by GPU 401 through the interface, and at the same time reads the full KV cache of the L-th decoder from the flash memory array 4022 (in Figure 7denoted by KVcacheL in Chinese), and then the L-th incremental KVcache is merged with the full KVcacheL in the vertical direction to obtain the merged full KVcache. The dimension of the merged full KVcache is 4×1536. According to q L , the merged full KVcache, and formula Execute the calculation of the second self-attention operator of the L-th decoder to obtain the L-th calculation result, and send the L-th calculation result to GPU401 through the interface;

[0114] GPU401 inputs the L-th calculation result into the feed-forward neural network layer of the L-th decoder. After being processed by the feed-forward neural network layer of the L-th decoder, the L-th output result of the L-th decoder can be obtained;

[0115] It should be noted here that as mentioned above, in order to improve the calculation efficiency in the embodiments of the present application, 8 GPUs 401 perform parallel calculations. That is, in the embodiments of the present application, the computing system includes 8 GPUs 401 and 1 SSD 402. Then, the calculations of the self-attention operators corresponding to each GPU 401 are all performed on this SSD 402. After the calculation, the SSD controller 4021 sends it to the corresponding GPU 401 through the interface. Then this GPU 401 will obtain the L-th output results calculated by other GPUs 401 and merge the 8 obtained L-th output results in the horizontal direction. As an example, the L-th output of the L-th decoder output by GPU1 is a 1×1536-dimensional vector, the L-th output of the L-th decoder output by GPU2 is a 1×1536-dimensional vector... The L-th output of the L-th decoder output by GPU8 is a 1×1536-dimensional vector. Then the merged vector is a 1×12288-dimensional vector. This GPU 401 infers the second output token based on the merged vector. Continuing with the above example, the second output token can be "town".

[0116] After determining the second output token, the second output token is used as the input sequence for this time. The subsequent process is the same as the process of inferring the output token "town" and will not be elaborated here until the last output token "monster in the river" is output, and the inference process ends.

[0117] Through the above technical solution, in the embodiment of the present application, during the inference process from the second output token to the last output token, in the inference of each output token, the calculation of the self-attention operator of each decoder involved is offloaded from the GPU to the SSD. In this way, only the incremental KV cache needs to be synchronized between the GPU and the SSD. Since the data volume of the incremental KV cache is much smaller than that of the full KV cache, the data transmission volume is reduced, the requirement for the transmission bandwidth of the SSD is lowered, and the real-time performance of the inference is improved.

[0118] Case 2

[0119] The above description is about the situation where the computing system includes 8 GPUs 401 and 1 SSD 402. To further improve the real-time performance of the inference, in the specific implementation process, multiple SSDs can be configured to carry the calculation and data transmission of the self-attention operator, that is, as shown above Figure 5 where the computing system includes 8 GPUs 501 and 8 SSDs 502. It was also mentioned above that when the computing system includes 8 GPUs 501 and 8 SSDs 502, the difference from the above Case 1 is only that in Case 1, after each GPU 401 calculates the full KV cache and incremental KV cache corresponding to each decoder, they are all sent to one SSD 402 included in the computing system. After the SSD controller 4021 finishes the calculation, it is sent to the corresponding GPU 401 through the interface. As an example, if the output tokens are "pagoda", "suppress", and "river demon", then the SSD controller 4021 needs to execute the calculation of the self-attention operator corresponding to each decoder sent by each GPU 401 during each token inference process. Here, since the 8 GPUs 501 are processed in parallel, the corresponding 8 SSDs 502 can also execute the calculation of the self-attention operator corresponding to each decoder sent by each GPU 501 in parallel, thereby improving the calculation efficiency and reducing the calculation time to improve the real-time performance of the inference. At the same time, the 8 SSDs jointly carry the transmission, which can further reduce the demand for the transmission bandwidth between the GPU and the SSD.

[0120] It should be noted here that the above descriptions are all based on the example where the architecture of the transformer only includes a decoder (Decoder only). Of course, in the specific implementation process, the calculation method of the self-attention operator provided in the present application can also be applied to the encoder-decoder architecture, which is not limited here.

[0121] Second, the embodiment of the present application also provides a method for calculating a self-attention operator, which is executed by an operator calculation device. The execution process can refer to the steps executed by the above SSD 402 and will not be elaborated here.

[0122] In a third aspect, an embodiment of the present application further provides a chip, including: a power supply circuit for power supply; a processor for obtaining a first incremental KV cache of a first decoder and a full KV cache of the first decoder among a plurality of decoders; and then performing calculation of a first self-attention operator of the first decoder according to the full KV cache and the first incremental KV cache of the first decoder to obtain a first calculation result and output the first calculation result.

[0123] In a fourth aspect, an embodiment of the present application further provides a computer program product, a computer program product including instructions, which when running on a computer, causes the computer to execute the following method: obtaining a first incremental KV cache of a first decoder and the full KV cache of the first decoder among a plurality of decoders; and then calculating a first self-attention operator of the first decoder according to the full KV cache and the first incremental KV cache of the first decoder to obtain a first calculation result and output the first calculation result.

[0124] As described above, the foregoing is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A computing system, characterized in that, It includes an inference device and an operator computing device. The inference device runs a Transformer model, and the Transformer model includes multiple decoders. The computing system includes: The inference device is configured to calculate a first incremental key-value cache (KVcache) of a first decoder among the multiple decoders based on a first output element token, and send the first incremental KVcache to the operator computing device, where the first output token is output by the inference device when performing the first token inference based on the received input sequence; The operator computing device is configured to perform the calculation of the first self-attention operator of the first decoder according to the full KVcache of the first decoder and the first incremental KVcache, obtain a first calculation result, and send the first calculation result to the inference device, where the full KVcache of the first decoder is pre-stored in the operator computing device.

2. The computing system according to claim 1, wherein: The inference device is further configured to determine a first output of the first decoder according to the first calculation result; The inference device is further configured to calculate a second incremental KVcache of a second decoder among the multiple decoders according to the first output, and send the second incremental KVcache to the operator computing device; The operator computing device is further configured to perform the calculation of the second self-attention operator of the second decoder according to the full KVcache of the second decoder and the second incremental KVcache, obtain a second calculation result, where the full KVcache of the second decoder is pre-stored in the operator computing device.

3. The computing system according to claim 1, wherein The computing system further includes other inference devices and other operator computing devices; The inference device is communicatively connected to the operator computing device, and the other inference devices are communicatively connected to the other operator computing devices; The operator computing device is configured to receive the first incremental KVcache sent by the inference device; The other operator computing devices are configured to receive other first incremental KVcaches sent by the other inference devices.

4. The computing system according to claim 1, wherein The inference device includes a graphics processing unit (GPU) or a general-purpose graphics processing unit (GPGPU).

5. The computing system according to claim 1, wherein The operator computing device includes a solid-state drive (SSD), a data processor (DPU), or a neural network processor (NPU).

6. The computing system according to claim 1, wherein The element is a token or an image patch.

7. A calculation method of a self-attention operator, characterized in that, Executed by an operator computing device, the computing method includes: Receiving the first incremental KVcache of the first decoder among the multiple decoders sent by an inference device, where the inference device runs a Transformer model, and the Transformer model includes multiple decoders; Perform the calculation of the first self-attention operator of the first decoder based on the full KV cache and the first incremental KV cache of the first decoder to obtain a first calculation result, where the full KV cache of the first decoder is pre-stored in the operator computing device, and the first calculation result is used to determine an output element token. Send the first calculation result to the inference device.

8. The calculation method according to claim 7, characterized in that, The method further includes: Receive the second incremental KV cache of the second decoder among the multiple decoders sent by the inference device, where the second incremental KV cache is calculated based on the first calculation result. Perform the calculation of the second self-attention operator of the second decoder according to the full KV cache and the second incremental KV cache of the second decoder, where the full KV cache of the second decoder is pre-stored in the operator computing device.

9. The calculation method according to claim 7, wherein The element is a token or an image patch.

10. An operator computing device, characterized in that, Includes: An interface and a controller. The interface is used to communicate with the inference device, and the inference device runs a Transformer model, which includes multiple decoders. The controller is configured to receive, through the interface, the first incremental KV cache of the first decoder among the multiple decoders. The controller is further configured to perform the calculation of the first self-attention operator of the first decoder according to the full KV cache and the first incremental KV cache of the first decoder to obtain a first calculation result, where the full KV cache of the first decoder is pre-stored in the operator computing device. The controller is further configured to send the first calculation result to the inference device through the interface.

11. The operator computing device according to claim 10, wherein The operator computing device further includes: A memory for storing the first incremental KV cache.

12. The operator computing device according to claim 10, characterized in that, The controller is further configured to: Receive, through the interface, the second incremental KV cache of the second decoder among the multiple decoders sent by the inference device, where the second incremental KV cache is calculated based on the first calculation result. Perform the calculation of the second self-attention operator of the second decoder according to the full KV cache and the second incremental KV cache of the second decoder, where the full KV cache of the second decoder is pre-stored in the operator computing device.

13. The operator calculation device according to claim 10, characterized in that, The controller includes a first processor and a second processor. The first processor is specifically configured to receive the first incremental KV cache through the interface. The second processor is specifically configured to perform the calculation of the first self-attention operator according to the full KV cache and the first incremental KV cache of the first decoder to obtain the first calculation result. The first processor is specifically further configured to send the first calculation result to the inference device through the interface.

14. The operator computing device according to claim 10, characterized in that, The operator computing device is a solid-state drive (SSD), a data processor unit (DPU), or a neural network processor unit (NPU).

15. The operator computing device according to claim 13, characterized in that, The second processor is a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).

16. A chip, characterized in that, It includes: A power supply circuit for power supply; A processor for obtaining a first incremental KV cache of a first decoder among a plurality of decoders and the full KV cache of the first decoder; According to the full KV cache and the first incremental KV cache of the first decoder, performing the calculation of a first self-attention operator of the first decoder to obtain a first calculation result, where the first self-attention operator is used to determine an output element token; Outputting the first calculation result.

Images