Method and device for processing data through self-attention model and related product

Abstract

The embodiment of the invention discloses a method and device for processing data through a self-attention model and a related product, and belongs to the technical field of AI. In the embodiment of the invention, for a scene in which the KV cache adopts a left filling mode to store data, the self-attention input parameters are expanded, so that the self-attention model can identify the effective data in the KV cache according to the expanded self-attention input parameters before performing self-attention calculation, and then the effective data not including redundant data are carried; on one hand, the amount of data carried from the KV cache region can be reduced, and on the other hand, redundant data can be prevented from participating in subsequent self-attention calculation, so that the self-attention calculation performance of the self-attention model can be greatly improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a method and device for processing data by using a self-attention model and related products. BACKGROUND

[0002] The self-attention model, also known as a Transformer model, is a neural network model based on a self-attention mechanism. Through the self-attention model, inference can be performed on the request data input by a user to obtain result data corresponding to the request data.

[0003] In related technologies, the self-attention model includes a self-attention operator, which is used to perform self-attention calculation based on KV data stored in a KV cache to predict the next element in the result data. Therefore, the self-attention operator needs to move the KV data from the KV cache before performing the self-attention calculation. In this scenario, the KV cache is filled with redundant data in addition to the KV data. In this scenario, how to improve the performance of the self-attention operator is a hot research topic. SUMMARY

[0004] Embodiments of the present application provide a method and device for processing data by using a self-attention model and related products, which can significantly improve the performance of the self-attention model in performing self-attention calculation in a scenario where the KV cache stores data using a left padding method. The technical solution is as follows:

[0005] In a first aspect, a method for processing data by using a self-attention model is provided, and the self-attention model corresponds to a key-value (KV) cache. In the method, request data of a user is obtained and input into the self-attention model. The result data corresponding to the request data is output by the self-attention model. In the process of determining each element in the result data, the self-attention model stores the KV data of the current element determined each time in a target partition of the KV cache. The idle area close to the starting position in the target partition is used to fill redundant data. The valid data excluding the redundant data is read from the target partition according to an extended self-attention input parameter. The valid data excluding the redundant data is subjected to self-attention calculation to determine the next element in the result data.

[0006] In the embodiments of the present application, for the scenario that the KV cache stores data in a left padding manner, the self-attention input parameter is expanded, so that the self-attention model can identify the effective data in the KV cache according to the expanded self-attention input parameter before performing self-attention calculation, and then carry the effective data that does not include redundant data. On the one hand, the amount of data carried from the KV cache area can be reduced, and on the other hand, the redundant data can be avoided from participating in the subsequent self-attention calculation, so that the performance of the self-attention model in performing self-attention calculation can be greatly improved.

[0007] In a possible implementation manner of the method provided in the first aspect, the result data includes prediction data, the target partition includes a first sub-partition close to the start position and a second sub-partition close to the end position, the first sub-partition and the second sub-partition each have a fixed size, the first sub-partition is used to store KV data of elements in the request data, the second sub-partition is used to store KV data of elements in the prediction data, and a free area close to the end position in the second sub-partition is used to fill redundant data; the expanded self-attention input parameter includes the end position of the target partition, an actual sequence length, and a padding length, the actual sequence length is an actual total length of the KV data stored in the target partition, and the padding length is a total length of the redundant data in the second sub-partition.

[0008] The above-mentioned manner is applied to a scenario in which the target partition is statically allocated. In this scenario, for different request data in a batch, the self-attention model allocates a target partition for each request data in the batch in the KV cache before processing the request data in the batch, and the target partitions corresponding to different request data are address-continuous. The target partition is not only used to store KV data of each element in the request data generated in the full-inference stage, but also used to store KV data of each element in the prediction data generated in the incremental-inference stage.

[0009] The first sub-partition is close to the start position of the target partition, and a free area close to the start position of the target partition is used to fill data, so that the KV data in the first sub-partition is stored in a left padding (i.e., right alignment) manner. The free area close to the end position in the second sub-partition is used to fill redundant data, so that the KV data in the second sub-partition is stored in a right padding (i.e., left alignment) manner. In this way, it can be ensured that the effective data in the first sub-partition and the effective data in the second sub-partition are continuous, so as to facilitate efficient reading of the effective data subsequently. In this scenario, in order to enable the self-attention model to identify the effective data in the KV cache according to the expanded self-attention input parameter, the expanded self-attention input parameter includes the end position of the target partition, an actual sequence length, and a padding length, the actual sequence length is an actual total length of the KV data stored in the target partition, and the padding length is a total length of the redundant data in the second sub-partition.

[0010] In a possible implementation manner of the method provided in the first aspect, the implementation process of reading the valid data excluding the redundant data from the target partition according to the extended self-attention input parameter can be: subtracting the actual sequence length and the padding length from the end position of the target partition in the extended self-attention input parameter to obtain the start position of the valid data in the target partition; subtracting the padding length from the end position of the target partition in the extended self-attention input parameter to obtain the end position of the valid data in the target partition; and reading the valid data excluding the redundant data from the target partition according to the start position of the valid data in the target partition and the end position of the valid data in the target partition.

[0011] The start position of the valid data in the target partition and the end position of the valid data in the target partition can be quickly determined in the scenario of statically allocating the target partition through the above formula.

[0012] In a possible implementation manner of the method provided in the first aspect, the implementation process of storing the KV data of the current element determined each time in the target partition of the KV cache by the self-attention model in the process of determining each element in the result data can be: after the KV data of the element in the request data is determined, storing the KV data of the element in the request data in a region close to the end position in the first sub-partition, and filling redundant data in the idle region in the first sub-partition and the second sub-partition, updating the actual sequence length to the total length of the KV data of the element in the request data, and updating the padding length to the total length of the second sub-partition; after the KV data of the i th element in the prediction data is determined, storing the KV data of the i th element in the second sub-partition at the storage position of the i th redundant data close to the start position in a way of overwriting, and updating the actual sequence length to the sum between the actual sequence length determined before the current time and the reference length, and updating the padding length to the difference between the actual sequence length determined before the current time and the reference length, the reference length being the length of the KV data of a single element.

[0013] Since the second sub-partition is gradually written with the KV data in the incremental inference stage, the actual sequence length and the padding length in the extended self-attention input parameter are also real-time changes.

[0014] In a possible implementation manner of the method provided in the first aspect, the result data includes prediction data, the target partition includes a first sub-partition close to the start position and a second sub-partition close to the end position, the first sub-partition has a fixed size, the first sub-partition is used to store KV data of elements in the request data, the second sub-partition is used to store KV data of elements in the prediction data, and the size of the second sub-partition is equal to the total length of the KV data of the elements in the prediction data generated before the current time; the extended self-attention input parameter includes the end position of the target partition and an actual sequence length, and the actual sequence length is the actual total length of the KV data stored in the target partition.

[0015] The above manner is applied to a scenario in which the target partition is dynamically allocated. In this scenario, for different request data in a batch, the self-attention model allocates a first sub-partition in the KV cache for each request data in the batch before processing the request data in the batch. The first sub-partition is used to store KV data of each element in the request data generated in the full-inference stage. After the KV data of the current element is generated for the first time in the subsequent incremental inference stage, the KV cache integrates the KV data of the current element and the data in the first sub-partition into a new target partition, and the target partition includes all data in the first sub-partition and the KV data of the current element. In the same way, after the KV data of the current element is generated for the i th time, the KV cache integrates the KV data of the current element and the data in the previous target partition into a new target partition, and the new target partition includes all data in the old target partition and the KV data of the current element. In this scenario, in order to enable the self-attention model to identify valid data in the KV cache according to the extended self-attention input parameter, the extended self-attention input parameter only needs to include the end position of the target partition and an actual sequence length, and the actual sequence length is the actual total length of the KV data stored in the target partition.

[0016] In a possible implementation manner of the method provided in the first aspect, the implementation process of reading valid data excluding redundant data from the target partition according to the extended self-attention input parameter can be: subtracting the actual sequence length from the end position of the target partition in the extended self-attention input parameter to obtain the start position of the valid data in the target partition; determining the end position of the target partition in the extended self-attention input parameter as the end position of the valid data in the target partition; and reading the valid data excluding redundant data from the target partition according to the start position of the valid data in the target partition and the end position of the valid data in the target partition.

[0017] Through the above formula, the start position of the valid data in the target partition and the end position of the valid data in the target partition can be quickly determined in the scenario in which the target partition is dynamically allocated.

[0018] In a possible implementation manner of the method provided in the first aspect, in the process of storing the KV data of the current element determined each time in the target partition of the KV cache by the self-attention model in determining the elements in the result data, the method can include the following steps: after the KV data of the element in the request data is determined, storing the KV data of the element in the request data in a region close to the end position in the first sub-partition, filling the idle region in the first sub-partition with redundant data, updating the end position of the target partition to the end position of the first sub-partition, and updating the actual sequence length to the total length of the KV data of the element in the request data; after the KV data of the i-th element in the prediction data is determined, sending the KV data of the i-th element to the KV cache, integrating all the data in the target partition and the KV data of the i-th element into the target partition of the current application by the KV cache, updating the end position of the target partition to the end position of the target partition of the current application, and updating the actual sequence length to the sum of the actual sequence length determined before the current time and the reference length, the reference length being the length of the KV data of a single element.

[0019] Since the KV data is gradually written in the incremental inference stage, the size of the second sub-partition gradually increases, and the actual sequence length in the extended self-attention input parameter also changes in real time accordingly.

[0020] In a possible implementation manner of the method provided in the first aspect, the self-attention calculation includes a plurality of intermediate calculations; for any intermediate calculation, the self-attention model performs the intermediate calculation according to effective data excluding redundant data, fills redundant data in the calculation result of the intermediate calculation according to the extended self-attention input parameter, and takes the filled calculation result as the output result of the intermediate calculation, so that the size of the output result is consistent with the size of the target partition.

[0021] When the self-attention model performs the self-attention calculation, it is necessary to ensure that the size of the output result of each intermediate calculation is consistent with the size of the target partition, so as to maintain the consistency of the data size. Therefore, after the self-attention model performs the intermediate calculation according to the effective data excluding the redundant data, it fills the redundant data in the calculation result of the intermediate calculation according to the extended self-attention input parameter, so that the size of the output result is consistent with the size of the target partition.

[0022] In a possible implementation manner of the method provided in the first aspect, the self-attention calculation is a fast self-attention (FA) calculation.

[0023] In the scenario where the self-attention operator is an FA operator, the performance bottleneck of the FA operator in the incremental inference stage is the memory access, that is, the performance bottleneck is the memory bound (the performance is limited by the bandwidth upper limit of the hardware itself). In this scenario, when the amount of data to be moved is reduced, the performance of the FA operator can be significantly improved.

[0024] In a second aspect, a device for processing data by a self-attention model is provided, which has functions to implement the method for processing data by a self-attention model in the first aspect. The device for processing data by a self-attention model comprises at least one module for implementing the method for processing data by a self-attention model provided in the first aspect.

[0025] In a third aspect, a computer device is provided, which comprises a processor and a memory in its structure. The memory is configured to store part or all of programs supporting the computer device to execute the method for processing data by a self-attention model provided in the first aspect, and store part or all of data involved in the method for processing data by a self-attention model provided in the first aspect. The processor is configured to execute the programs stored in the memory.

[0026] In a fourth aspect, a computer readable storage medium is provided, which stores instructions that, when executed on a computer, cause the computer to perform the method for processing data by a self-attention model in the first aspect.

[0027] In a fifth aspect, a computer program product is provided, which contains instructions that, when executed on a computer, cause the computer to perform the method for processing data by a self-attention model in the first aspect.

[0028] The technical effects obtained by the corresponding technical means in the second aspect to the fifth aspect are similar, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a schematic diagram of a right padding mode and a left padding mode provided by an embodiment of the present application;

[0030] Figure 2 is an architecture schematic diagram of an AI computing system provided by an embodiment of the present application;

[0031] Figure 3 is an architecture schematic diagram of an encoder of a certain layer in a self-attention model provided by an embodiment of the present application;

[0032] Figure 4 is a flow schematic diagram of a self-attention calculation of a self-attention model supporting a fusion core provided by an embodiment of the present application;

[0033] Figure 5 is a time consumption comparison schematic diagram of a self-attention calculation provided by an embodiment of the present application;

[0034] Figure 6 is a method flowchart for processing data by a self-attention model provided by an embodiment of the present application;

[0035] Figure 7 is a schematic diagram of a target partition provided by an embodiment of the present application;

[0036] Figure 8 is a schematic diagram of another target partition provided by an embodiment of the present application;

[0037] Figure 9 is a schematic diagram of filling redundant data to a calculation result of intermediate calculation provided by an embodiment of the present application;

[0038] Figure 10 is a schematic diagram of filling redundant data to a calculation result of intermediate calculation provided by another embodiment of the present application;

[0039] Figure 11 is a schematic diagram of filling redundant data to a calculation result of intermediate calculation provided by another embodiment of the present application;

[0040] Figure 12 is a calculation flowchart of a self-attention model supporting a fusion core provided by an embodiment of the present application;

[0041] Figure 13 is a structural schematic diagram of an apparatus for processing data by a self-attention model provided by an embodiment of the present application;

[0042] Figure 14 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0044] Before the embodiments of the present application are explained in detail, the application scenarios of the embodiments of the present application will be explained.

[0045] The self-attention model (also known as the Transformer model) is designed to process sequence data, which refers to a set of data arranged in a certain order. Compared with traditional recurrent neural networks (RNN), the self-attention model has better parallel performance and shorter training time, and can more effectively handle long-distance dependencies. In addition, positional encoding is introduced in the self-attention model to overcome the problem that the self-attention mechanism ignores the positions of elements in the sequence data. Therefore, the self-attention model has achieved significant performance improvement in many tasks such as natural language processing, machine translation, text summarization, and question answering systems, and has become an important part of natural language processing research and application.

[0046] For the convenience of subsequent description, the sequence data input to the self-attention model is referred to as the request data (also known as the prompt), and the sequence data output by the self-attention model based on the request data is referred to as the result data (also known as the result). Among them, the result data includes the prediction data, which refers to the sequence data predicted by the self-attention model based on the request data. For example, the request data is "What is your name?", and the prediction data is "The name is ×××". Optionally, the result data includes not only the prediction data but also the request data. For example, the request data is "What is your name?", and the result data is "What is your name? The name is ×××".

[0047] The self-attention model includes an encoder-only model, a decoder-only model, and an encoder+decoder model. The encoder-only model refers to a model that includes an encoder but does not include a decoder in its construction, the decoder-only model refers to a model that includes a decoder but does not include an encoder in its construction, and the encoder+decoder model refers to a model that includes both an encoder and a decoder in its construction.

[0048] The inference process of the self-attention model on the request data includes the following two stages:

[0049] The full-inference phase (also known as the prefill phase): In the full-inference phase, the self-attention model needs to calculate the self-attention of the entire request data through the self-attention operator, and the full-inference phase only needs to be calculated once. The self-attention model subsequently infers the first element (token) in the prediction data based on the self-attention of the entire request data. Therefore, the self-attention calculation in the full-inference phase can also be referred to as full self-attention calculation.

[0050] In addition, in the full reasoning phase, the sizes of q_input, k_input, and v_input input to the self-attention operator are all [seq_len, emb_dim], seq_len is the sequence length of the request data, representing that the Query (Q) data, Key (K) data, and Value (V) data input to the self-attention operator in the full reasoning phase are Q data, K data, and V data of all elements included in the request data, emb_dim is the size of the embedding vector of a single element, and the embedding vector of each element includes a word embedding vector of the element and a position embedding vector of the element, the word embedding vector is used to indicate the semantic features of the element, and the position embedding vector is used to indicate the position features of the element in the request data. The Q data, K data, and V data can be understood as matrices obtained by performing different linear transformations on the embedding vectors of the elements.

[0051] The generation phase: the generation phase is also referred to as the decoding phase, in the generation phase, each time an element in the predicted data is generated, the self-attention between the element and all previous elements needs to be calculated once through the self-attention operator, the previous elements include the request data and the predicted data reasoned before the current time. Therefore, the self-attention calculation in the generation phase can also be referred to as the incremental self-attention calculation.

[0052] In addition, in the generation phase, the size of q_input input to the self-attention operator is [1, emb_dim], “1” represents that the Q data input to the self-attention operator is the Q data of the element (referred to as the current element) in the predicted data generated last time, and emb_dim is also the size of the embedding vector of each element. The sizes of k_input and v_input input to the self-attention operator are [n, emb_dim], n represents that the K data and V data input to the self-attention operator are the K data and V data of the current element and all previous elements, and emb_dim is also the size of the embedding vector of each element. In other words, in the generation phase, the Q data of the current element, the K data and V data of the current element and all previous elements are input to the self-attention operator to calculate the self-attention between the current element and all previous elements, and then to reason the next element in the predicted data.

[0053] In the incremental inference stage, the self-attention operator needs to calculate the self-attention of the current element and all the previous elements, and therefore needs to first calculate the K data and V data of the current element and all the previous elements. However, the K data and V data of all the previous elements have been calculated before the current time, and therefore can be saved in a certain cache as two tensors with a size of [seq_len-1, inner_dim], one for storing the K data and the other for storing the V data, where seq_len-1 is the total length of all the previous elements, and inner_dim is the size of the embedding vector of a single element, which is the same as the emb_dim described above.

[0054] Since the K data and V data of all the previous elements have been saved in a certain cache, only the K data and V data of the current element need to be calculated before each round of incremental self-attention calculation, thereby improving the performance of the self-attention model.

[0055] In the embodiments of the present application, the cache is referred to as a KV cache. In other words, the self-attention model corresponds to a KV cache.

[0056] In some scenarios, the KV cache corresponding to the self-attention model can be managed by the following three schemes. It should be noted that the KV cache is used to store the K data and V data, and the K data and V data are stored separately in the same way. For ease of description, the management of data in the KV cache is described below by taking KV data as an example, where the KV data refers to the K data or the V data.

[0057] Scheme one: static batching mode. For a batch, a memory block with a continuous address in the KV cache is allocated. Since a batch includes multiple request data, for each request data, a partition with a continuous address is allocated in the memory block according to the maximum output length. In the incremental inference process for a certain request data, the KV data of the current element is scattered to the partition corresponding to the request data. In this scenario, the free area in the partition is filled with redundant data to convert the discontinuous or unequal length data in adjacent partitions into a continuous tensor.

[0058] Among them, there are two ways to fill the redundant data in the partition, which are right padding mode and left padding mode. Figure 1 is a schematic diagram of the right padding mode and the left padding mode provided by the embodiments of the present application. Figure 1One row represents one partition allocated for one request data, and different rows represent partitions allocated for different request data in the same batch. Each block represents an address block of KV data used to store a single element in the partition. The block with diagonal lines represents that the corresponding address block stores valid KV data, and the white block represents that the corresponding address block is filled with invalid redundant data.

[0059] As shown in FIG. 3, the right padding mode can be understood as follows: the KV data is stored along the starting position of the partition, that is, stored in the left-justified manner. After the KV data is stored, if there is a free area near the end position in the partition, the free area is filled with redundant data. The left padding mode can be understood as follows: the KV data is stored along the end position of the partition, that is, stored in the right-justified manner. After the KV data is stored, if there is a free area near the starting position in the partition, the free area is filled with redundant data. Figure 1

[0060] Solution two: dynamic batch processing mode. For any request data, the memory used by the KV data generated in the incremental inference stage is applied on demand. For example, after the new K _i V _ i data is generated, the new K _i V _ i data and the previously cached KV data are concatenated to a partition with continuous addresses by the KV cache, which can improve the utilization of the KV cache. In this scenario, in the full-inference stage, for different request data, a fixed partition can be directly allocated to store the KV data generated in the full-inference stage. At this time, the free area in the partition is also filled with redundant data.

[0061] Solution three: page attention mode. The valid data in the KV cache is managed by a virtual page table. The valid data refers to the data in the KV cache other than the redundant data. The redundant data is also referred to as invalid data. In this scenario, there is no redundant data in the KV cache. However, because the virtual page table is introduced, the calculation performance of the self-attention model will be affected to a certain extent. Therefore, the page attention mode is generally used in scenarios with long input sequences or limited system memory to improve the throughput.

[0062] In addition, in some scenarios, the self-attention model is by default in the left padding mode.

[0063] ​For the above-mentioned scheme one and scheme two, if the left padding mode is used to store the KV data in the KV cache. When the subsequent self-attention operator carries the KV data from the KV cache, the self-attention input parameters provided by the self-attention model to the self-attention operator usually include the start position and the end position of the partition corresponding to the current request data in the KV cache, causing the self-attention operator to be unable to carry the effective data excluding the redundant data through the self-attention input parameters, that is, the data carried by the self-attention operator from the KV cache includes the redundant data, so that these redundant data will participate in the subsequent self-attention calculation, thereby causing the performance of the self-attention operator to be seriously degraded.

[0064] Based on this, the embodiment of the present application provides a method for processing data through a self-attention model. In the method, for the scenario that the KV cache stores data in the left padding mode, the self-attention input parameters are expanded, so that the self-attention model can identify the effective data in the KV cache according to the expanded self-attention input parameters before performing self-attention calculation, and then carry the effective data excluding the redundant data, so that the performance of the self-attention model in performing self-attention calculation can be greatly improved.

[0065] The system architecture related to the embodiment of the present application is explained and described below.

[0066] Figure 2 is a schematic diagram of an AI computing system provided by the embodiment of the present application. The AI computing system is used to realize the function of the self-attention model in the embodiment of the present application.

[0067] As shown in Figure 2 , the AI computing system includes a framework layer, a compiling layer and an executing layer.

[0068] Among them, the framework layer is used to provide an application programming interface (API) that can be called by a user to realize AI computing related composition. The framework layer can be realized by MINDSPORE, TENSORFLOW, PYTORCH and the like AI framework, for example.

[0069] The compiling layer is configured to implement intermediate representation (IR) conversion to complete the compilation and loading of the self-attention model. The compiling layer includes, for example, a graph preparing module, a graph slicing module, a graph optimizing module, a graph compiling module, and a graph loading module.

[0070] The execution layer is configured to distinguish various operator types through task scheduling, and then complete the execution of various operators in the self-attention model and output results. The execution layer includes, for example, a runtime module, a task scheduler, a graphics processing unit (GPU) operator, a natural processing unit (NPU) operator, and a central processing unit (CPU) operator.

[0071] In the embodiments of the present application, the self-attention input parameters of the self-attention operator running in the execution layer are expanded to optimize the performance of the self-attention operator. The type of the self-attention operator can be a GPU operator, an NPU operator, a CPU operator, and the like, which is not limited in the embodiments of the present application. The type of the self-attention operator depends on the type of the request data input to the self-attention model. For example, if the request data is image data, the type of the self-attention operator is a GPU operator. For example, if the request data is text data, the type of the self-attention operator is a CPU operator.

[0072] The structure of the self-attention model involved in the embodiments of the present application is explained below.

[0073] The self-attention model encodes the request data and generates result data through a multi-layer encoder and / or a multi-layer decoder structure. Each layer of the encoder or the decoder includes a self-attention operator and a feedforward neural network operator, and uses residual connection and layer normalization techniques to accelerate model convergence and improve performance.

[0074] The structure of the encoder and the decoder is basically the same, and the structure of the self-attention model is described below by taking the encoder in the self-attention model as an example.

[0075] Figure 3 is a schematic diagram of an encoder of a certain layer in a self-attention model provided by the embodiments of the present application. As shown inFigure 3 As shown, the layer encoder is marked as Nx, Nx includes a multi-head attention operator and a position-wise feedforward neural network (position-wise FNN) operator, a residual connection and a layer normalization (i.e. Add&Norm) operator.

[0076] The multi-head attention operator can be understood as connecting multiple self-attention operators in series to implement multiple self-attention calculations, and the number of heads n in the multi-head attention operator is also the number of times of self-attention calculation.

[0077] Among them, the algorithm adopted in the self-attention operator is the self-attention mechanism (Self-Attention Mechanism). The calculation process of the self-attention mechanism includes three steps: calculating attention weights (i.e. the importance of each element to other elements), calculating self-attention scores, and generating the output of the self-attention operator.

[0078] For example, the self-attention mechanism can be represented by the following formula:

[0079] Softmax(Q*K T )*V

[0080] Among them, Q, K and V represent Q data, K data and V data input to the self-attention operator, respectively.

[0081] For example, the code of the self-attention mechanism is as follows:

[0082] def attention(q_input, k_input, v_input):

[0083] q = self.Q(q_input)

[0084] k = self.K(k_input)

[0085] v = self.V(v_input)

[0086] return Softmax(q * k.transpose()) * v

[0087] Figure 3 The internal calculation flowchart of one of the self-attention operators of the multi-head self-attention operator is also shown in the figure. As shown in the figure, Figure 3As shown, the input of the self-attention operator is Q data, K data and V data. The self-attention operator performs matrix multiplication (MatMul, MM) calculation on the input Q data and the transpose of K data to obtain attention weights (i.e., Q*K T ), then performs Softmax calculation on the attention weights to obtain self-attention scores, and finally performs MM calculation on the self-attention scores and V data to obtain the output of the self-attention operator.

[0088] Figure 3 The self-attention operator in the self-attention model is implemented by a matrix calculation (cube) kernel in the processor such as a GPU to perform MM calculation, by a vector calculation (vector) kernel in the processor to perform Softmax calculation, and by other small operators (not shown in the self-attention model) to perform calculation. Figure 3 The other small operators can include a mask operator and a dropout operator, which is a strategy to solve model overfitting. In other words, Figure 3 Different calculations inside the self-attention operator in the self-attention model are implemented by different kernels in the processor, which results in the calculation results of each kernel being cached in high bandwidth memory (HBM) for further calculation by other kernels.

[0089] The memory of the processor is composed of a plurality of sub-memories with different sizes and different read / write speeds. For example, the memory of the processor includes SRAM, HBM (also known as video memory), and DRAM, where the read / write speed of SRAM is about 19 TB / s, the read / write speed of HBM is only 1.5 TB / s, which is less than 1 / 10 of SRAM, and the read / write speed of DRAM is lower than that of HBM. Therefore, the main bottleneck of the above self-attention operator is memory access, so it is very important to reduce the read / write times of the GPU to the HBM in the self-attention model and effectively use the faster SRAM for calculation.

[0090] Moreover, Figure 3 The calculation complexity and space complexity of the self-attention model shown are both O(N 2 ), where N is the sequence length of the request data. This results in the calculation amount and storage complexity of the self-attention model increasing quadratically with the sequence length N of the request data, which limits the maximum sequence length of the request data of the self-attention model.

[0091] The self-attention model supporting fusion kernels can reduce the calculation complexity and space complexity to O(N). The self-attention operator in this scenario is also called a flash attention (FA) operator.

[0092] Figure 4 This is a schematic diagram illustrating the process of self-attention calculation using a self-attention model that supports a fusion kernel, as provided in an embodiment of this application. Figure 4 As shown, in this FA operator, the FA slicing algorithm is used to slice the input Q, K, and V, and the Q, K, and V of each slice are directly processed by a fused kernel. Figure 3 The self-attention operator shown in the figure is used to perform all the calculations. Finally, the calculation results of Q, K and V of different slices are summarized to obtain the output of the FA operator.

[0093] Since it can be accomplished using a fusion core Figure 3 The self-attention operator shown essentially integrates the cube core and vector core of the processor into a fused core. This eliminates the need to cache intermediate calculation results such as Q*K and Softmax in the HBM, thus reducing the number of HBM reads and writes during self-attention calculations and consequently saving end-to-end (E2E) time. Performance is significantly improved compared to... Figure 3 The self-attention operator shown can improve performance by more than 2 times.

[0094] Figure 5 This is a schematic diagram comparing the time consumption of self-attention calculation provided in an embodiment of this application. For example... Figure 5 As shown, in Figure 3 In the self-attention operator shown, different kernels are used to implement different intermediate calculations in the self-attention process, such as MM, masking, softmax, and model optimization strategies. Dropout is used as an example of a model optimization strategy. Figure 4 In the FA operator shown, all intermediate calculations in the self-attention computation process can be achieved through a single fusion kernel, therefore the time consumption of one self-attention computation is only a fraction of the time required. Figure 4 The self-attention calculation shown takes about 10% of the time.

[0095] In addition, Figure 3 In the self-attention operator shown, because the input data is not sliced, each intermediate computation result consumes too much storage resources in HBM. For example, with a batch size (B) of 10, a head number (N) of 40, and the shapes (S) of Q, K, and V of 4096, and a data type fp16, approximately 6.25 GB of memory is needed to store the intermediate computation results. And... Figure 4In the illustrated FA operator, because the input data is sliced, the intermediate calculation results such as Q*K and Softmax are only the intermediate calculation results corresponding to the slices, that is, the intermediate calculation results of the blocks. These intermediate calculation results of the blocks can be directly saved in the local memory, thereby reducing the memory consumption. The local memory can be understood as the local memory of a chip implementing the fusion kernel.

[0096] Based on this, in the embodiments of the present application, the self-attention operator in the self-attention model can be an FA operator to further improve the performance of the self-attention model. In other words, in the embodiments of the present application, the implementation manner of the self-attention calculation by the self-attention model can be that the FA calculation is performed by the self-attention model.

[0097] In addition, in the embodiments of the present application, the self-attention model can be any type of self-attention model, such as a large language model (LLM). Here, no longer one by one example is illustrated.

[0098] The method of processing data by the self-attention model provided in the embodiments of the present application is explained in detail below.

[0099] Figure 6 is a method flowchart of processing data by a self-attention model provided in the embodiments of the present application. As Figure 6 shown, the method includes the following steps.

[0100] Step 601: Obtain the request data of a user, and input the request data to the self-attention model.

[0101] Step 602: Output the result data corresponding to the request data by the self-attention model.

[0102] In the process of determining each element in the result data, the self-attention model stores the KV data of the current element determined each time in the target partition of the KV cache. The idle area close to the starting position in the target partition is used to fill the redundant data, and the effective data excluding the redundant data is read from the target partition according to the extended self-attention input parameter. The effective data excluding the redundant data is calculated by the self-attention to determine the next element in the result data.

[0103] In the target partition, the idle area close to the starting position is used to fill the redundant data, that is, the target partition stores data in the left padding manner.

[0104] In the embodiment of the present application, for the scenario that the KV cache stores data in a left padding manner, the self-attention input parameter is expanded to enable the self-attention model to identify valid data in the KV cache according to the expanded self-attention input parameter before performing self-attention calculation, and then to carry valid data that does not include redundant data. On the one hand, this can reduce the amount of data carried from the KV cache area, and on the other hand, it can also avoid redundant data from participating in subsequent self-attention calculation, thus greatly improving the performance of the self-attention model in performing self-attention calculation.

[0105] The implementation manner in which the self-attention model reads valid data that does not include redundant data from the target partition according to the expanded self-attention input parameter can be as follows: the self-attention model determines the start position and the end position of the valid data in the target partition according to the expanded self-attention input parameter, and reads the valid data that does not include redundant data from the target partition according to the determined start position and end position of the valid data.

[0106] In addition, the target partition can be understood as a storage space in the KV cache for storing KV data generated in the process of processing the request data. Based on the foregoing, it can be known that the target partition can be statically allocated or dynamically allocated, and therefore the expanded self-attention input parameter will be explained and described below taking these two scenarios as examples.

[0107] Scenario one: the target partition is statically allocated.

[0108] In scenario one, for different request data in a batch, the self-attention model allocates a target partition in the KV cache for each request data in the batch before processing the request data in the batch, and the target partitions corresponding to different request data are address-continuous. The target partition is not only used to store the KV data of each element in the request data generated in the full-inference stage, but also used to store the KV data of each element in the prediction data generated in the incremental-inference stage.

[0109] For example, in scenario one, the target partition includes a first sub-partition close to the start position and a second sub-partition close to the end position, the sizes of the first sub-partition and the second sub-partition are both fixed, the first sub-partition is used to store the KV data of the elements in the request data, the second sub-partition is used to store the KV data of the elements in the prediction data, and the idle area close to the end position in the second sub-partition is used to fill redundant data.

[0110] The first sub-partition is close to the start position of the target partition, and the free area close to the start position of the target partition is used to fill data, so that the KV data in the first sub-partition is stored in a left padding (that is, right alignment) manner. The free area close to the end position in the second sub-partition is used to fill redundant data, so that the KV data in the second sub-partition is stored in a right padding (that is, left alignment) manner. In this way, it can be ensured that the effective data in the first sub-partition and the effective data in the second sub-partition are continuous, so as to facilitate subsequent efficient reading of the effective data.

[0111] Figure 7 is a schematic diagram of a target partition provided by an embodiment of the present application. Figure 7 Each row in represents a target partition for a request data, Figure 7 Each block in represents an address block of KV data for storing a single element in the target partition, and the oblique line block represents that valid KV data is stored in the corresponding address block, and the white block represents that invalid redundant data is filled in the corresponding address block.

[0112] As shown in Figure 7 , the KV data in the first sub-partition is stored in a left padding manner, so if the length of the KV data of all elements of the request data is less than the length of the first sub-partition, the free area close to the start position in the first sub-partition is filled with redundant data. The second sub-partition is used to store the KV data of each element in the predicted data calculated in the incremental inference stage. Since the incremental inference stage infers each element in the predicted data one by one, the KV data of each element in the incremental inference stage is also written to the second sub-partition step by step.

[0113] For example, for the target partition of the first row, since the length of the KV data of all elements of the requested data is less than the length of the first sub-partition, there is padding data on the left side of the first sub-partition. And before the incremental inference stage, the KV data of the first element in the predicted data has not been determined, so before the incremental inference stage, the second sub-partition is filled with redundant data. After entering the incremental inference stage, before the first incremental inference calculation, the KV data of the first element in the predicted data needs to be determined first, and the determined KV data of the current element is stored in the storage location corresponding to the first block on the left side of the second sub-partition, and then all the valid data in the target partition is read, and a self-attention calculation is performed according to the read valid data to determine the second element in the predicted data. By analogy, before the i-th incremental inference calculation, i is greater than 1, the KV data of the i-th element in the predicted data needs to be determined first, and the determined KV data of the current element is stored in the storage location corresponding to the i-th block on the left side of the second sub-partition, and then all the valid data in the target partition is read, and a self-attention calculation is performed according to the read valid data to determine the i+1-th element in the predicted data.

[0114] Based on this, in scenario one, in order to enable the self-attention model to identify the valid data in the KV cache according to the extended self-attention input parameter, the extended self-attention input parameter includes the end position of the target partition, the actual sequence length, and the padding length. The actual sequence length is the actual total length of the KV data stored in the target partition, and the padding length is the total length of the redundant data in the second sub-partition.

[0115] Correspondingly, the implementation manner of reading the valid data excluding the redundant data from the target partition according to the extended self-attention input parameter can be: subtracting the actual sequence length and the padding length from the end position of the target partition in the extended self-attention input parameter to obtain the start position of the valid data in the target partition; subtracting the padding length from the end position of the target partition in the extended self-attention input parameter to obtain the end position of the valid data in the target partition; and reading the valid data excluding the redundant data from the target partition according to the start position of the valid data in the target partition and the end position of the valid data in the target partition.

[0116] Suppose the actual sequence length is marked as act_seq_len, the padding length is marked as padding_size, and the end position of the target partition is marked as Smax. The start position and the end position of the valid data in the target partition can be represented by the following formulas:

[0117] The start position of the valid data in the target partition = Smax-padding_size-act_seq_len;

[0118] End position of valid data in target partition = Smax - padding_size.

[0119] The end position of the target partition can be obtained from a KV shape parameter of the KV cache. In other words, the KV shape parameter is stored in the self-attention model, and the end position of the target partition corresponding to each requested data is recorded in the KV shape parameter.

[0120] Since the second sub-partition is gradually written with KV data in the incremental inference stage, the actual sequence length and the padding length in the extended self-attention input parameter are also real-time changes. Based on this, in some embodiments, in the process of determining each element in the result data, the self-attention model stores the KV data of the current element determined each time in the target partition of the KV cache in the following manner: in the full-inference stage, after determining the KV data of an element in the requested data, the KV data of the element in the requested data is stored in the area close to the end position in the first sub-partition, the free area in the first sub-partition is filled with redundant data, and the actual sequence length is updated to the total length of the KV data of the element in the requested data, and the padding length is updated to the total length of the second sub-partition.

[0121] Correspondingly, in the incremental inference stage, after determining the KV data of the i-th element in the predicted data, the KV data of the i-th element is stored in the storage position of the i-th redundant data close to the start position in the second sub-partition in an overwriting manner, and the actual sequence length is updated to the sum between the actual sequence length determined before the current time and the reference length, and the padding length is updated to the difference between the actual sequence length determined before the current time and the reference length, and the reference length is the length of the KV data of a single element.

[0122] For example, the reference length is 1, then in the incremental inference stage, after storing the KV data of a current element in the KV cache each time, the actual sequence length is +1 to obtain the updated actual sequence length, and the padding length is -1 to obtain the updated padding length.

[0123] Scenario two: the target partition is dynamically allocated.

[0124] In the second scenario, for different request data in a batch, the self-attention model allocates a first sub-partition for each request data in the batch in the KV cache before processing the request data in the batch. The first sub-partition is used only to store the KV data of each element in the request data generated in the full inference stage. In the subsequent incremental inference stage, after the KV data of the current element is generated for the first time, the KV cache integrates the KV data of the current element and the data in the first sub-partition into a new target partition, and the target partition includes all the data in the first sub-partition and the KV data of the current element. In the same way, after the KV data of the current element is generated for the i-th time, the KV cache integrates the KV data of the current element and the data in the previous target partition into a new target partition, and the new target partition includes all the data in the old target partition and the KV data of the current element.

[0125] Based on this, in the second scenario, the target partition includes a first sub-partition close to the starting position and a second sub-partition close to the ending position, the size of the first sub-partition is a fixed size, the first sub-partition is used to store the KV data of the elements in the request data, and the second sub-partition is used to store the KV data of the elements in the predicted data, and the size of the second sub-partition is equal to the total length of the KV data of the elements in the predicted data generated before the current time.

[0126] Figure 8 is another schematic diagram of a target partition provided by an embodiment of the present application. Figure 8 Each row in represents a target partition for a request data, Figure 8 Each block in represents an address block for storing the KV data of a single element in the target partition, and the oblique line block represents that the corresponding address block stores valid KV data, and the white block represents that the corresponding address block is filled with invalid redundant data.

[0127] As shown in Figure 8 , the KV data in the first sub-partition is stored in a left padding manner, so if the length of the KV data of all elements in the request data is less than the length of the first sub-partition, the left side of the first sub-partition is filled with redundant data. The second sub-partition is used to store the KV data of each element in the predicted data calculated in the incremental inference stage, and the size of the second sub-partition is dynamically applied based on the size of the KV data generated in the incremental inference stage, so the size of the second sub-partition is consistent with the size of the KV data generated in the incremental inference stage, and therefore the second sub-partition has no redundant data.

[0128] For example, for the target partition in the first row, the length of the key-value data of all elements in the requested data is less than the length of the first sub-partition. In this case, the left side of the first sub-partition has padding data. Furthermore, before the incremental inference phase, the key-value data of the first element in the prediction data has not yet been determined; therefore, the target partition only contains the first sub-partition. After entering the incremental inference phase, before the first incremental inference calculation, the key-value data of the first element in the prediction data needs to be determined. The determined key-value data of the current element and the data in the first sub-partition are then integrated into a single target partition. At this point, the target partition includes both the first and second sub-partitions. The size of the second sub-partition is the size of the key-value data of the current element. Subsequently, all valid data is read from the target partition, and a self-attention calculation is performed based on the read valid data to determine the second element in the prediction data. Similarly, before the i-th incremental inference calculation, i is greater than 1. It is necessary to first determine the KV data of the i-th element in the prediction data, and integrate the determined KV data of the current element with all the data in the previous target partition into a new target partition. Then, all the valid data is read from the new target partition, and a self-attention calculation is performed based on the read valid data to determine the (i+1)-th element in the prediction data.

[0129] Based on this, in Scenario 2, in order for the self-attention model to identify valid data in the KV cache according to the extended self-attention input parameters, the extended self-attention input parameters include the end position of the target partition and the actual sequence length, where the actual sequence length is the actual total length of the KV data stored in the target partition.

[0130] Accordingly, the implementation method for reading valid data excluding redundant data from the target partition based on the extended self-attention input parameters can be as follows: subtract the actual sequence length from the end position of the target partition in the extended self-attention input parameters to obtain the start position of valid data in the target partition; determine the end position of the target partition in the extended self-attention input parameters as the end position of valid data in the target partition; and read valid data excluding redundant data from the target partition based on the start position and the end position of valid data in the target partition.

[0131] like Figure 8 As shown, assuming the actual sequence length is labeled as `act_seq_len` and the end position of the target partition is labeled as `Smax`, the start and end positions of valid data in the target partition can be represented by the following formulas:

[0132] The starting position of valid data in the target partition = Smax - act_seq_len;

[0133] The end position of valid data in the target partition = Smax.

[0134] Since the KV data is written gradually in the incremental inference stage, the size of the second sub-partition also gradually increases, and the actual sequence length in the extended self-attention input parameter also changes in real time accordingly. Based on this, in some embodiments, in the process of determining each element in the result data, the self-attention model stores the KV data of the current element each time in the target partition of the KV cache in the following manner: in the full inference stage, after determining the KV data of the element in the request data, the KV data of the element in the request data is stored in the area close to the end position in the first sub-partition, redundant data is filled in the idle area in the first sub-partition, and the end position of the first sub-partition is updated to the end position of the target partition, and the actual sequence length is updated to the total length of the KV data of the element in the request data.

[0135] Correspondingly, in the incremental inference stage, after determining the KV data of the i-th element in the prediction data, i is greater than or equal to 1, the KV data of the i-th element is sent to the KV cache, the KV cache integrates all the data in the target partition and the KV data of the i-th element into the currently applied target partition, and the end position of the target partition is updated to the end position of the currently applied target partition, and the actual sequence length is updated to the sum between the actual sequence length determined before the current time and the reference length, and the reference length is the length of the KV data of a single element.

[0136] For example, the reference length is 1, then in the incremental inference stage, after sending the KV data of a current element to the KV cache each time, the KV cache integrates the KV data of the current element and all the data in the previous target partition into a new currently applied target partition, then the actual sequence length is updated by 1 to obtain the updated actual sequence length, and the end position of the target partition is updated to the end position of the currently applied target partition.

[0137] In addition, after the KV cache integrates all the data in the target partition and the KV data of the i-th element into the currently applied target partition, the KV cache also deletes all the data in the original target partition to release the storage space of the original target partition.

[0138] In the above scenario one and scenario two, when the self-attention model carries data from the KV cache, it can only carry valid data, so the amount of data carried is reduced, thereby improving the performance of the self-attention operator in the self-attention model.

[0139] Moreover, in the scenario where the self-attention operator is an FA operator, the performance bottleneck of the FA operator in the incremental inference stage is the memory access, that is, the performance bottleneck is the memory bound (the performance is limited by the bandwidth upper limit of the hardware itself), and in this scenario, when the amount of data carried is reduced, the performance of the FA operator can be significantly improved.

[0140] In addition, after the self-attention model moves the valid data (excluding redundant data) from the KV cache, for any intermediate computation among multiple intermediate computations, the self-attention model performs intermediate computation based on the valid data (excluding redundant data), and fills the redundant data in the computation result of the intermediate computation according to the extended self-attention input parameters. The filled computation result is used as the output result of the intermediate computation so that the data size of the output result is consistent with the size of the target partition.

[0141] This avoids redundant data from participating in the intermediate computation processes of self-attention, further improving the performance of the self-attention operator.

[0142] When performing self-attention calculations, the self-attention model needs to ensure that the size of the output of each intermediate calculation is consistent with the size of the target partition to maintain data size consistency. Therefore, after performing intermediate calculations based on valid data excluding redundant data, the self-attention model also fills the intermediate calculation results with redundant data according to the extended self-attention input parameters to make the size of the output result consistent with the size of the target partition.

[0143] For example, self-attention computation includes a first MM computation, a softmax computation, and a second MM computation.

[0144] For the first MM calculation, the transpose of the Q data of the current element and the K data moved from the KV cache (i.e., K) is used. T Perform MM calculations to obtain the result Q*K of the first MM calculation. T .like Figure 9 As shown, the slashed squares represent valid data, and the white squares represent invalid data. The process involves transposing the Q data of the current element and the K data moved from the KV cache (i.e., K...). T When performing the first MM calculation, only valid data participates in the calculation, and the result obtained at this time is only Q*K. T The corresponding diagonal squares represent valid data. In this scenario, the calculation results are filled according to the size of the target partition corresponding to the K data, such as... Figure 9 As shown, the target partition corresponding to K data is filled with two redundant data on the left and right sides. Therefore, two redundant data are filled on the left and right sides of the calculation result to obtain the output result of the first MM calculation.

[0145] For the Softmax calculation, the result of the first MM calculation is subjected to Softmax calculation to obtain the Softmax calculation result Q*K. T .like Figure 10As shown, the slanted squares represent valid data, and the white squares represent invalid data. The result of the first MM calculation, Q*K, is then used. T When performing Softmax, only valid data participates in the calculation, and the result obtained is only Softmax(Q*K). T The valid data represented by the diagonal squares corresponding to K. In this scenario, the calculation results of Softmax are filled according to the size of the target partition corresponding to K data, such as... Figure 9 As shown, the target partition corresponding to data K is filled with two redundant data points on both the left and right sides, therefore, as Figure 10 As shown, two redundant data points are filled on the left and right sides of the Softmax calculation result to obtain the output result of the Softmax calculation.

[0146] For the second MM calculation, the result of the Softmax calculation and the V data moved from the KV cache are used to perform the MM calculation, resulting in the second MM calculation result Softmax(Q*KT)*V. For example... Figure 11 As shown, the diagonally marked squares represent valid data, and the white squares represent invalid data. This is used when performing MM calculations using the results of the Softmax calculation and the V data retrieved from the KV cache. Figure 11 The V data in the table represents the V data after changing the axis position; that is, each column represents the target partition corresponding to a requested data, and only valid data participates in the calculation. For example... Figure 11 As shown, since the size of the calculation result obtained from the second MM calculation is consistent with the size of the target partition corresponding to the V data, there is no need to fill redundant data in the calculation result. In other words, the calculation result obtained from the second MM calculation is also the output result of the second MM calculation.

[0147] The following example illustrates the method provided in the embodiments of this application using a self-attention model as the supporting fusion kernel.

[0148] Figure 12 This is a schematic diagram of the computational flow of a self-attention model supporting a fusion kernel, provided in an embodiment of this application. For example... Figure 12 As shown, the self-attention model supporting the fusion kernel includes two types of FA operators: full FA operator and incremental FA operator. The full FA operator is used to perform FA calculation in the full inference stage, and the incremental FA operator is used to perform FA calculation in the incremental inference stage.

[0149] like Figure 12As shown, the KV data of all elements in the request data generated in the full reasoning stage is saved to the KV cache in a left padding manner, and the Q data, K data and V data of all elements in the request data are calculated by the full FA operator. Then in the incremental reasoning stage, after the KV data of an element in the prediction data is generated each time, the KV data of the current element is saved to the KV cache, and then all the KV data of the elements generated before the current time (excluding redundant data) is carried from the KV cache, and incremental FA calculation is performed by the incremental FA operator to infer the next element in the prediction data.

[0150] To sum up, in the embodiments of the present application, for the scenario that the KV cache stores data in a left padding manner, the self-attention input parameter is expanded to enable the self-attention model to identify the valid data in the KV cache according to the expanded self-attention input parameter before performing self-attention calculation, and then carry the valid data excluding redundant data, which can reduce the amount of data carried from the KV cache area on the one hand, and can also avoid redundant data from participating in subsequent self-attention calculation, thus greatly improving the performance of the self-attention model in performing self-attention calculation.

[0151] Figure 13 is a structural schematic diagram of a device for processing data by a self-attention model provided by an embodiment of the present application, and the self-attention model corresponds to a key-value (KV) cache; as Figure 13 shown, the device 1300 includes the following modules.

[0152] The obtaining module 1301 is configured to obtain the request data of a user and input the request data to the self-attention model; for details, refer to step 601 in the Figure 6 embodiment.

[0153] The output module 1302 is configured to output the result data corresponding to the request data by the self-attention model; for details, refer to step 602 in the Figure 6 embodiment.

[0154] In the process of determining each element in the result data, the self-attention model stores the KV data of the current element determined each time in the target partition of the KV cache, the idle area close to the starting position in the target partition is used to fill redundant data, and the valid data excluding redundant data is read from the target partition according to the expanded self-attention input parameter, and the valid data excluding redundant data is calculated by the self-attention model to determine the next element in the result data.

[0155] Optionally, the result data includes predicted data, the target partition includes a first sub-partition close to the start position and a second sub-partition close to the end position, sizes of the first sub-partition and the second sub-partition are fixed sizes, the first sub-partition is configured to store KV data of elements in the request data, the second sub-partition is configured to store KV data of elements in the predicted data, and a free area close to the end position in the second sub-partition is configured to fill redundant data.

[0156] In this scenario, the extended self-attention input parameter includes an end position of the target partition, an actual sequence length, and a padding length, the actual sequence length is an actual total length of KV data stored in the target partition, and the padding length is a total length of redundant data in the second sub-partition.

[0157] Optionally, the self-attention model is configured to: subtract the actual sequence length and the padding length from the end position of the target partition in the extended self-attention input parameter to obtain a start position of effective data in the target partition; subtract the padding length from the end position of the target partition in the extended self-attention input parameter to obtain an end position of the effective data in the target partition; and read the effective data excluding the redundant data from the target partition according to the start position of the effective data in the target partition and the end position of the effective data in the target partition.

[0158] Optionally, the self-attention model is further configured to: after determining the KV data of the elements in the request data, store the KV data of the elements in the request data in a region close to the end position in the first sub-partition, fill the redundant data in the free area in the first sub-partition and the second sub-partition, update the actual sequence length to a total length of the KV data of the elements in the request data, and update the padding length to a total length of the second sub-partition; after determining the KV data of the i th element in the predicted data, store the KV data of the i th element in the second sub-partition at a storage position of the i th redundant data close to the start position in an overwrite manner, update the actual sequence length to a sum between a determined actual sequence length before the current time and a reference length, update the padding length to a difference between the determined actual sequence length before the current time and the reference length, and the reference length is a length of KV data of a single element.

[0159] Optionally, the result data includes predicted data, the target partition includes a first sub-partition close to the start position and a second sub-partition close to the end position, a size of the first sub-partition is a fixed size, the first sub-partition is configured to store KV data of elements in the request data, the second sub-partition is configured to store KV data of elements in the predicted data, and a size of the second sub-partition is equal to a total length of KV data of elements in the predicted data generated before the current time.

[0160] In this scenario, the extended self-attention input parameter includes an end position of the target partition and an actual sequence length, where the actual sequence length is an actual total length of the KV data stored in the target partition.

[0161] Optionally, the self-attention model is configured to: subtract the actual sequence length from the end position of the target partition in the extended self-attention input parameter to obtain a start position of valid data in the target partition; determine the end position of the target partition in the extended self-attention input parameter as an end position of the valid data in the target partition; and read the valid data excluding the redundant data from the target partition according to the start position of the valid data in the target partition and the end position of the valid data in the target partition.

[0162] Optionally, the self-attention model is configured to: after determining the KV data of the element in the request data, store the KV data of the element in the request data in a region close to the end position in the first sub-partition, fill the redundant data in the free area of the first sub-partition, and update the end position of the target partition to the end position of the first sub-partition and update the actual sequence length to the total length of the KV data of the element in the request data; and after determining the KV data of the i-th element in the predicted data, send the KV data of the i-th element to the KV cache, integrate all the data in the target partition and the KV data of the i-th element into the currently applied target partition by the KV cache, and update the end position of the target partition to the end position of the currently applied target partition and update the actual sequence length to the sum of the actual sequence length determined before the current time and a reference length, where the reference length is the length of the KV data of a single element.

[0163] Optionally, the self-attention calculation includes a plurality of intermediate calculations, and for any intermediate calculation, the self-attention model is configured to: perform the intermediate calculation according to the valid data excluding the redundant data, fill the redundant data in the calculation result of the intermediate calculation according to the extended self-attention input parameter, and take the filled calculation result as the output result of the intermediate calculation, so that the size of the output result is consistent with the size of the target partition.

[0164] Optionally, the self-attention calculation is a fast self-attention (FA) calculation.

[0165] In summary, in the embodiments of the present application, for the scenario where the KV cache stores data in a left padding manner, the self-attention input parameter is extended, so that the self-attention model can identify the valid data in the KV cache according to the extended self-attention input parameter before performing the self-attention calculation, and then carry the valid data excluding the redundant data. On the one hand, the amount of data carried from the KV cache area can be reduced, and on the other hand, the redundant data can be prevented from participating in the subsequent self-attention calculation, so that the performance of the self-attention model in performing the self-attention calculation can be greatly improved.

[0166] It should be noted that the device for processing data by the self-attention model provided in the above embodiments is only exemplified by the above division of functional modules when processing data by the self-attention model. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for processing data by the self-attention model provided in the above embodiments and the method for processing data by the self-attention model belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0167] Figure 14 is a structural schematic diagram of a computer device provided by an embodiment of the present application. The AI computing system in the foregoing embodiments can be implemented by one or more computer devices shown in the figure. Figure 14 Referring to FIG. 14, Figure 14 The computer device includes at least one processor 1401, a communication bus 1402, a memory 1403, and at least one communication interface 1404.

[0168] The processor 1401 can be a general central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application scheme.

[0169] The communication bus 1402 can include a path for transmitting information between the above components.

[0170] Memory 1403 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 1403 may exist independently and be connected to processor 1401 via communication bus 1402. Memory 1403 may also be integrated with processor 1401.

[0171] The memory 1403 stores program code for executing the scheme of this application, and its execution is controlled by the processor 1401. The processor 1401 executes the program code stored in the memory 1403. The program code may include one or more software modules. The self-attention models in the foregoing embodiments can determine the data used for application development through the processor 1401 and one or more software modules in the program code in the memory 1403.

[0172] Communication interface 1404 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0173] In a specific implementation, as one example, a computer device may include multiple processors, for example... Figure 14 The processors 1401 and 1405 are shown. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0174] In a specific implementation, as an example, the computer device can further include an output device 1406 and an input device 1407. The output device 1406 is in communication with the processor 1401 and can display information in a variety of manners. For example, the output device 1406 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 1407 is in communication with the processor 1401 and can receive user input in a variety of manners. For example, the input device 1407 can be a mouse, a keyboard, a touch screen device, a sensor device, or the like.

[0175] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of computer device.

[0176] In the embodiments described above, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example: floppy disk, hard disk, magnetic tape), an optical medium (for example: digital versatile disc (DVD)), or a semiconductor medium (for example: solid state disk (SSD)), etc.

[0177] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0178] The above is not intended to limit the embodiments of the present application, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method for processing data using a self-attention model, characterized in that, The self-attention model corresponds to a key-value cache; the method includes: Obtain user request data and input the request data into the self-attention model; The self-attention model outputs the result data corresponding to the requested data. In the process of determining each element in the result data, the self-attention model stores the key-value (KV) data of the current element in the target partition of the KV cache. The free area near the starting position in the target partition is used to fill redundant data. The model reads valid data excluding redundant data from the target partition according to the extended self-attention input parameters, and performs self-attention calculation on the valid data excluding redundant data to determine the next element in the result data.

2. The method as described in claim 1, characterized in that, The result data includes prediction data. The target partition includes a first sub-partition near the start position and a second sub-partition near the end position. The size of the first sub-partition and the second sub-partition is fixed. The first sub-partition is used to store the key-value data of the elements in the request data, and the second sub-partition is used to store the key-value data of the elements in the prediction data. The free area near the end position in the second sub-partition is used to fill redundant data. The extended self-attention input parameters include the end position of the target partition, the actual sequence length, and the padding length. The actual sequence length is the actual total length of the KV data stored in the target partition, and the padding length is the total length of the redundant data in the second sub-partition.

3. The method as described in claim 2, characterized in that, The step of reading valid data (excluding redundant data) from the target partition based on the extended self-attention input parameters includes: Subtract the actual sequence length and the padding length from the end position of the target partition in the extended self-attention input parameters to obtain the starting position of the valid data in the target partition; Subtract the padding length from the end position of the target partition in the extended self-attention input parameters to obtain the end position of the valid data in the target partition; Based on the start and end positions of the valid data in the target partition, read valid data (excluding redundant data) from the target partition.

4. The method as described in claim 2 or 3, characterized in that, In determining each element in the result data, the self-attention model stores the key-value (KV) data of the current element at each determination in the target partition of the KV cache, including: After determining the key-value data of the elements in the requested data, the key-value data of the elements in the requested data is stored in the region near the end position in the first sub-partition, and redundant data is filled in the free region in the first sub-partition and the second sub-partition. The actual sequence length is updated to the total length of the key-value data of the elements in the requested data, and the filling length is updated to the total length of the second sub-partition. After determining the KV data of the i-th element in the predicted data, the KV data of the i-th element is stored in the storage location of the i-th redundant data near the starting position in the second sub-partition using an overwrite write method. The actual sequence length is updated to the sum of the actual sequence length determined before the current time and the reference length, and the padding length is updated to the difference between the actual sequence length determined before the current time and the reference length, where the reference length is the length of the KV data of a single element.

5. The method as described in claim 1, characterized in that, The result data includes prediction data. The target partition includes a first sub-partition near the start position and a second sub-partition near the end position. The size of the first sub-partition is fixed. The first sub-partition is used to store the key-value data of the elements in the request data. The second sub-partition is used to store the key-value data of the elements in the prediction data. The size of the second sub-partition is equal to the total length of the key-value data of the elements in the prediction data generated before the current time. The extended self-attention input parameters include the end position of the target partition and the actual sequence length, where the actual sequence length is the actual total length of the KV data stored in the target partition.

6. The method as described in claim 5, characterized in that, The step of reading valid data (excluding redundant data) from the target partition based on the extended self-attention input parameters includes: Subtract the actual sequence length from the end position of the target partition in the extended self-attention input parameters to obtain the start position of the valid data in the target partition; The end position of the target partition in the extended self-attention input parameters is determined as the end position of the valid data in the target partition; Based on the start and end positions of the valid data in the target partition, read valid data (excluding redundant data) from the target partition.

7. The method as described in claim 5 or 6, characterized in that, In determining each element in the result data, the self-attention model stores the key-value (KV) data of the current element at each determination in the target partition of the KV cache, including: After determining the key-value data of the elements in the requested data, the key-value data of the elements in the requested data is stored in the region near the end position of the first sub-partition, redundant data is filled in the free region of the first sub-partition, the end position of the target partition is updated to the end position of the first sub-partition, and the actual sequence length is updated to the total length of the key-value data of the elements in the requested data. After determining the KV data of the i-th element in the predicted data, the KV data of the i-th element is sent to the KV cache. The KV cache integrates all the data in the target partition and the KV data of the i-th element into the currently applied target partition, updates the end position of the target partition to the end position of the currently applied target partition, and updates the actual sequence length to the sum of the actual sequence length determined before the current time and the reference length, where the reference length is the length of the KV data of a single element.

8. The method according to any one of claims 1-7, characterized in that, The self-attention calculation includes multiple intermediate calculations; For any intermediate calculation, the self-attention model performs the intermediate calculation based on valid data excluding redundant data, and fills the calculation result of the intermediate calculation with redundant data according to the extended self-attention input parameters. The filled calculation result is used as the output result of the intermediate calculation, so that the size of the output result is consistent with the size of the target partition.

9. The method according to any one of claims 1-8, characterized in that, The self-attention calculation is an accelerated self-attention FA calculation.

10. An apparatus for processing data using a self-attention model, characterized in that, The self-attention model corresponds to a key-value (KV) cache; The device includes: The acquisition module is used to acquire user request data and input the request data into the self-attention model; The output module is used to output the result data corresponding to the request data through the self-attention model; In the process of determining each element in the result data, the self-attention model stores the key-value (KV) data of the current element in the target partition of the KV cache. The free area near the starting position in the target partition is used to fill redundant data. The model reads valid data excluding redundant data from the target partition according to the extended self-attention input parameters, and performs self-attention calculation on the valid data excluding redundant data to determine the next element in the result data.

11. The apparatus as claimed in claim 10, characterized in that, The result data includes prediction data. The target partition includes a first sub-partition near the start position and a second sub-partition near the end position. The size of the first sub-partition and the second sub-partition is fixed. The first sub-partition is used to store the key-value data of the elements in the request data, and the second sub-partition is used to store the key-value data of the elements in the prediction data. The free area near the end position in the second sub-partition is used to fill redundant data. The extended self-attention input parameters include the end position of the target partition, the actual sequence length, and the padding length. The actual sequence length is the actual total length of the KV data stored in the target partition, and the padding length is the total length of the redundant data in the second sub-partition.

12. The apparatus as claimed in claim 11, characterized in that, The self-attention model is used for: Subtract the actual sequence length and the padding length from the end position of the target partition in the extended self-attention input parameters to obtain the starting position of the valid data in the target partition; Subtract the padding length from the end position of the target partition in the extended self-attention input parameters to obtain the end position of the valid data in the target partition; Based on the start and end positions of the valid data in the target partition, read valid data (excluding redundant data) from the target partition.

13. The apparatus as claimed in claim 11 or 12, characterized in that, The self-attention model is also used for: After determining the key-value data of the elements in the requested data, the key-value data of the elements in the requested data is stored in the region near the end position in the first sub-partition, and redundant data is filled in the free region in the first sub-partition and the second sub-partition. The actual sequence length is updated to the total length of the key-value data of the elements in the requested data, and the filling length is updated to the total length of the second sub-partition. After determining the KV data of the i-th element in the predicted data, the KV data of the i-th element is stored in the storage location of the i-th redundant data near the starting position in the second sub-partition using an overwrite write method. The actual sequence length is updated to the sum of the actual sequence length determined before the current time and the reference length, and the padding length is updated to the difference between the actual sequence length determined before the current time and the reference length, where the reference length is the length of the KV data of a single element.

14. The apparatus as claimed in claim 10, characterized in that, The result data includes prediction data. The target partition includes a first sub-partition near the start position and a second sub-partition near the end position. The size of the first sub-partition is fixed. The first sub-partition is used to store the key-value data of the elements in the request data. The second sub-partition is used to store the key-value data of the elements in the prediction data. The size of the second sub-partition is equal to the total length of the key-value data of the elements in the prediction data generated before the current time. The extended self-attention input parameters include the end position of the target partition and the actual sequence length, where the actual sequence length is the actual total length of the KV data stored in the target partition.

15. The apparatus as claimed in claim 14, characterized in that, The self-attention model is used for: Subtract the actual sequence length from the end position of the target partition in the extended self-attention input parameters to obtain the start position of the valid data in the target partition; The end position of the target partition in the extended self-attention input parameters is determined as the end position of the valid data in the target partition; Based on the start and end positions of the valid data in the target partition, read valid data (excluding redundant data) from the target partition.

16. The apparatus as claimed in claim 14 or 15, characterized in that, The self-attention model is used for: After determining the key-value data of the elements in the requested data, the key-value data of the elements in the requested data is stored in the region near the end position of the first sub-partition, redundant data is filled in the free region of the first sub-partition, the end position of the target partition is updated to the end position of the first sub-partition, and the actual sequence length is updated to the total length of the key-value data of the elements in the requested data. After determining the KV data of the i-th element in the predicted data, the KV data of the i-th element is sent to the KV cache. The KV cache integrates all the data in the target partition and the KV data of the i-th element into the currently applied target partition, updates the end position of the target partition to the end position of the currently applied target partition, and updates the actual sequence length to the sum of the actual sequence length determined before the current time and the reference length, where the reference length is the length of the KV data of a single element.

17. The apparatus according to any one of claims 10-16, characterized in that, The self-attention calculation includes multiple intermediate calculations; For any intermediate computation, the self-attention model is used to: perform the intermediate computation based on valid data excluding redundant data, and fill the computation result of the intermediate computation with redundant data according to the extended self-attention input parameters, and use the filled computation result as the output result of the intermediate computation, so that the size of the output result is consistent with the size of the target partition.

18. The apparatus as described in any one of claims 10-17, characterized in that, The self-attention calculation is an accelerated self-attention FA calculation.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1-9.

20. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-9.

Images