Attention mechanism operation method, equipment and medium
By introducing fusion operators, the linear transformation and position coding in the attention mechanism are integrated into one operation, which solves the problem of inefficient operation in the existing technology, realizes efficient parallel processing, and improves computing efficiency and model performance.
Patent Information
- Application Number
- CN202411216233.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-22
AI Technical Summary
The existing attention mechanism is inefficient in operation in large language models, mainly because linear transformation and position encoding need to be implemented through independent operators, resulting in redundant memory operations and multiple kernel startups, which cannot implement parallel processing, resulting in failure to obtain computing resources. Make full use of it.
Introduce fusion operators to integrate the linear transformation and position encoding of queries, keys, and values into a unified operation, reduce the number of kernel startups, avoid frequent parameter loading, and perform position encoding while linear transformation of values to realize linear transformation and position Parallel processing of encoding.
This greatly improves the computing efficiency, avoids calculation bottlenecks, improves the operating efficiency of the attention mechanism, and improves the overall performance of the model.
Smart Images

Figure CN119005249A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operation method, device and medium of an attention mechanism. Background Art
[0002] In the design of large language models, the attention mechanism is one of the core components. The attention mechanism first performs a linear transformation on the input data to generate query, key, and value vectors, and positionally encodes the query vector and key vector. Next, the similarity score between the query vector and the key vector is calculated and normalized. Finally, the value vector is weighted and summed to generate the attention output.
[0003] In the current implementation method of the attention mechanism, both linear transformation and position encoding need to be implemented through their respective corresponding operators. In order to perform linear transformation of query, key, and value to calculate the query vector, key vector, and value vector, the input data needs to be loaded three times repeatedly, resulting in redundant memory operations and time overhead. In addition, the operation of different operators needs to be implemented through multiple kernel launches, which adds additional performance overhead and further reduces computational efficiency.
[0004] In addition, for AI chips composed of heterogeneous computing cores, the tensor computing core (Tcore) is responsible for matrix multiplication and accumulation (MMA) to achieve linear transformation, and the vector computing core (Vcore) is responsible for position encoding. Since the tasks corresponding to these computing cores are executed sequentially in the calculation of the attention mechanism, parallel operations cannot be achieved, which makes the computing resources not fully utilized.
[0005] Therefore, how to optimize the operation of the attention mechanism is an urgent problem to be solved. Summary of the invention
[0006] The present invention provides an operation method, device and medium of an attention mechanism, so as to solve the defect of low operation efficiency of the attention mechanism in the related art.
[0007] The present invention provides an operation method of an attention mechanism, comprising the following steps: Based on the input data, start the fusion operator of the attention mechanism; Based on the fusion operator, query linear transformation, key linear transformation and value linear transformation are performed on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0008] According to a method for operating an attention mechanism provided by the present invention, the step of sequentially performing a query linear transformation, a key linear transformation, and a value linear transformation on the input data includes: Loading query weights, key weights and value weights required for performing the query linear transformation, key linear transformation and value linear transformation; Based on the query weight, key weight and value weight, query linear transformation, key linear transformation and value linear transformation are sequentially performed on the input data.
[0009] According to an operation method of an attention mechanism provided by the present invention, based on the query weight, key weight and value weight, the input data is sequentially subjected to query linear transformation, key linear transformation and value linear transformation, including: Dividing the query weight, the key weight, and the value weight into sub-weights respectively; The sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight are distributed to each processing unit, and each processing unit is controlled to perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence based on the sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight.
[0010] According to an operation method of an attention mechanism provided by the present invention, the number of the sub-weights is consistent with the number of the processing units.
[0011] According to an operation method of an attention mechanism provided by the present invention, when the query weight, key weight and value weight are quantized weights, it also includes: Load scaling parameters; Before position encoding the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation, scaling the query vector and the key vector based on the scaling parameter; And, after obtaining the value vector obtained by the linear transformation of the value, scaling the value vector based on the scaling parameter.
[0012] According to an operation method of an attention mechanism provided by the present invention, when the query weight, key weight and value weight are quantized weights, it also includes: Based on the fusion operator, the scaled value vector is reconstructed.
[0013] According to an operation method of an attention mechanism provided by the present invention, position encoding of the query vector obtained by the linear transformation of the query and the key vector obtained by the linear transformation of the key includes: Load position encoding parameters; Based on the position encoding parameters, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an operating method of the attention mechanism as described above is implemented.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an operating method of the attention mechanism as described in any one of the above.
[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements an operating method of any of the attention mechanisms described above.
[0017] The operation method, device and medium of the attention mechanism provided by the present invention integrate the independent calculation steps in the traditional attention mechanism into a unified operation by introducing a fusion operator, thereby reducing the number of kernel startups and avoiding frequent and repeated parameter loading, thereby greatly improving the calculation efficiency. In addition, during the operation of the fusion operator, position encoding is performed while the value is linearly transformed, realizing parallel processing of linear transformation and position encoding, avoiding the calculation bottleneck caused by sequential execution in traditional methods, and making the operation process of the attention mechanism more efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flowchart of the attention mechanism in the related art.
[0020] Figure 2 It is one of the flow charts of the operation method of the attention mechanism provided by the present invention.
[0021] Figure 3 It is a schematic diagram of the weight shuffling technology in the fusion operator provided by the present invention.
[0022] Figure 4 It is a schematic diagram of the calculation flow of the fusion operator provided by the present invention.
[0023] Figure 5This is the second flow chart of the operation method of the attention mechanism provided by the present invention.
[0024] Figure 6 It is a structural schematic diagram of the operating device of the attention mechanism provided by the present invention.
[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] In the design of large language models, the attention mechanism is an indispensable core component. By introducing linear transformations of queries, keys, and values, as well as positional encodings for query vectors and key vectors, the attention mechanism can effectively capture the correlation in the input data and ensure that the model can focus on key information when processing long sequences. This mechanism greatly improves the performance of the model and achieves significant results in natural language processing tasks. Therefore, optimizing the computational process in the attention mechanism is of great significance to improving the performance of the entire model.
[0028] In the current implementation of the attention mechanism, both linear transformation and position encoding need to be implemented through their respective operators. For example, Figure 1 This is a flowchart of the attention mechanism in the related art. Figure 1 As shown in the figure, first, the feature input is used as input and passes through three independent linear transformations (i.e., query linear transformation, key linear transformation, and value linear transformation) to generate three vectors: query vector, key vector, and value vector. This process is completed through matrix multiplication operations, and the specific calculation process is: Q=mma(input, Q_weight), K=mma(input, K_weight), V=mma(input, V_weight). Among them, mma( ) represents matrix multiplication, Q, K, and V represent query, key, and value vectors respectively, input is the input feature, and Q_weight, K_weight, and V_weight represent the weight matrices of query, key, and value respectively. For artificial intelligence chips composed of heterogeneous computing cores, such as GPUs (Graphics Processing Units), the above linear transformation process can be implemented by tensor computing cores in GPUs.
[0029] After the query vector, key vector, and value vector are calculated, the query vector and key vector are then subjected to Rotary Position Embedding (ROPE) operation. In the attention mechanism, ROPE is only applied to the query vector and key vector, not to the value vector. This is because the main goal of the attention mechanism is to capture the positional relationship and correlation in the input data through the similarity between the query and key vectors. The positional relationship refers to the relative positional relationship between tokens (the basic unit formed after the text in the input data is processed after word segmentation) in the input data, for example, the order relationship between a token and other tokens in a sentence; the correlation refers to the semantic or content correlation between different tokens in the input data. For example, the degree of semantic or contextual association between one token and another token. The role of the value vector is to provide the specific feature content of the input data for these correlations, that is, detailed information about each token in the input data, such as word meaning, part of speech, or contextual information. Suppose the input data is a sentence: "The cat sat on the mat". The query vector and key vector are used to capture the correlation between each token and other tokens in the sequence. For example, the word "cat" may have a strong correlation with the word "sat" because they are directly related in the sentence. The value vector contains the specific features of the word "cat", such as its meaning (animal), part of speech (noun), and its context in the sentence. Therefore, positional encoding is not required for the value vector.
[0030] By introducing position information into the query vector and key vector through rotational position encoding, the attention mechanism can identify the correlation between words at different positions in the input data. For example, when two words (i.e., the basic unit after word segmentation, which may be a complete word or a part of a word) have a certain distance relationship in the input data, the rotational position encoding enables the dot product of the query vector and the key vector to reflect this position relationship, thereby affecting the calculation of the attention weight. The attention weight represents the importance of each word in the input data in the final output. The rotational position encoding adjusts the dot product results of the query and key vectors so that the model can more accurately capture and reflect the relative position relationship between these words, thereby adjusting their proportion in the attention weight. For artificial intelligence chips composed of heterogeneous computing cores, such as GPUs, rotational position encoding can be completed by the vector computing cores in the GPU.
[0031] In addition, the key vector and the value vector can be cached, that is, the key cache and the value cache are obtained for subsequent autoregressive generation tasks.
[0032] From the above process, we can see that the current attention mechanism has some obvious shortcomings in its implementation. First, linear transformation and position encoding need to be implemented through their respective operators. Therefore, when calculating the query vector, key vector, and value vector, the input data needs to be loaded separately and then independent matrix multiplication operations need to be performed. That is, the input data needs to be loaded three times in this process, which leads to the repetitiveness of data loading and the redundancy of memory operations. In addition, the operation of different operators needs to be implemented through multiple kernel startups, which adds additional performance overhead and further reduces the computational efficiency.
[0033] In addition, since the tensor computing core and vector computing core of the GPU are responsible for query, key, value linear transformation and rotational position encoding respectively, the calculation process of the attention mechanism is divided into sequential execution and parallel processing cannot be achieved, resulting in low utilization of computing resources.
[0034] In response to the above problems, an embodiment of the present invention provides an operation method of an attention mechanism. Figure 2 This is one of the flow charts of the operation method of the attention mechanism provided by the present invention. Figure 2 As shown, the method can be applied to artificial intelligence chips, for example, it can be applied to GPU, or it can also be applied to GPGPU (General-Purpose computing on Graphics Processing Unit), NPU (Neural network Processing Unit), DPU (Deep learning Processing Unit), APU (Accelerated Processing Unit), etc., and the embodiments of the present invention do not make specific limitations on this.
[0035] The method includes: Step 210, based on the input data, start the fusion operator of the attention mechanism.
[0036] Specifically, for a model including an attention mechanism, during the model reasoning and training process, a feature matrix output by a network layer above the attention layer in the model can be received, which is recorded as input data in the embodiment of the present invention. The input data can be the result of processing by a multi-layer neural network, which can include high-dimensional vector information. It can be understood that the input data is the input of the attention mechanism.
[0037] In order to solve the problem of low efficiency of attention mechanism in related technologies, the concept of fusion operator is introduced in the embodiment of the present invention. Different from the traditional attention mechanism, the linear transformation of query, key, value and rotation position encoding are all operated in the form of independent operators, where the operation of each operator requires loading the parameters required for its own operation and starting the corresponding kernel. The fusion operator aims to integrate these independent operators into a unified computing task, that is, to fuse these independent operators into one operator and run them as a whole.
[0038] In the above fusion operator, the linear transformation of query, key, and value vectors is first implemented, as well as the position encoding of query vectors and key vectors. Through this integration method, the fusion operator can complete all the steps of the above attention mechanism at one time, and because the fusion operator itself is used as an operator, only one kernel startup is required, avoiding the overhead of multiple startup calculation tasks. In addition, since multiple calculation operations can be completed in one fusion operator, it is only necessary to load the required parameters once when the fusion operator is started, avoiding the situation where each operation needs to load data separately. This design not only reduces the frequency of memory access and saves memory bandwidth, but also further improves the overall performance of the attention mechanism and even the model operation.
[0039] Step 220, based on the fusion operator, perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0040] Specifically, the fusion operator is an operator that integrates the linear transformation and rotational position encoding of the query, key, and value into a unified computing task in step 210. During the operation of the fusion operator, the query linear transformation, key linear transformation, and value linear transformation can be performed on the input data in sequence, thereby obtaining a query vector, a key vector, and a value vector, respectively. In addition, during the operation of the fusion operator, position encoding operations can also be performed on the query vector and the key vector.
[0041] In the related art, it is usually necessary to wait until the query, key, and value vectors are all generated before performing the subsequent position encoding operation; in the embodiment of the present invention, the fusion operator can perform the position encoding operation at the same time as the linear transformation of the value vector by introducing a parallel processing mechanism. That is, during the process of performing the linear transformation of the value, the fusion operator begins to perform position encoding on the query vector and key vector that have been generated. This parallel operation allows the query vector and key vector to enter the position encoding stage immediately after they are generated, without having to wait for the transformation of the value vector to be completed.
[0042] For example, during the operation of the fusion operator, the tensor computing core on the artificial intelligence chip can perform query linear transformation, key linear transformation and value linear transformation in sequence, and after the query linear transformation and key linear transformation are completed and the query vector and key vector are obtained, that is, when the value linear transformation starts to run, the vector computing core on the artificial intelligence chip can perform position encoding for the query vector and the key vector, that is, while the tensor computing core is performing value linear transformation, the vector computing core can perform position encoding, and both types of computing cores are running in the same time period, so the computing resources can be fully utilized, and the operating efficiency of the attention mechanism can be further improved.
[0043] The operation method of the attention mechanism provided by the embodiment of the present invention integrates the independent calculation steps in the traditional attention mechanism into a unified operation by introducing a fusion operator, reduces the number of kernel startups, avoids frequent and repeated parameter loading, and thus greatly improves the calculation efficiency. In addition, during the operation of the fusion operator, position encoding is performed while the value is linearly transformed, realizing parallel processing of linear transformation and position encoding, avoiding the calculation bottleneck caused by sequential execution in traditional methods, and making the operation process of the attention mechanism more efficient.
[0044] Based on the above embodiment, in step 220, the query linear transformation, key linear transformation and value linear transformation are sequentially performed on the input data, including: Loading query weights, key weights and value weights required for performing the query linear transformation, key linear transformation and value linear transformation; Based on the query weight, key weight and value weight, query linear transformation, key linear transformation and value linear transformation are sequentially performed on the input data.
[0045] Specifically, the fusion operator first loads the required query weights, key weights, and value weights together. Query weights, key weights, and value weights are the parameters used in the attention mechanism to convert input data into query vectors, key vectors, and value vectors. These weights are usually obtained through model training, and their role is to map the high-dimensional features of the input into the vector space used for attention calculation. At this stage, by loading all the necessary weights at once, the fusion operator can avoid the redundant memory operations and additional overhead caused by loading different weights in batches in the traditional method, thereby improving the efficiency of data loading.
[0046] Next, based on the loaded query weights, key weights, and value weights, the input data can be subjected to query linear transformation, key linear transformation, and value linear transformation in turn. Specifically, the query linear transformation is to multiply the input data with the query weight matrix to generate a query vector, which is used to find relevance in the subsequent attention mechanism; the key linear transformation is to multiply the input data with the key weight matrix to generate a key vector, which is used to match the query vector to determine the relevance of each word in the input data; the value linear transformation is to multiply the input data with the value weight matrix to generate a value vector, which contains the actual information content in the input data and will be used for weighted summation after the attention weight is calculated.
[0047] In the method provided by the embodiment of the present invention, by loading the weights of the query, key, and value at one time, the fusion operator can complete the linear transformation operation of the input data in a more efficient manner, avoiding the time delay and resource waste caused by the gradual loading and conversion in the traditional method. This optimization not only improves the consistency and efficiency of data processing, but also ensures that in a complex model, each linear transformation step can be closely coordinated to maximize the overall performance of the model.
[0048] Based on the above embodiment, the step of sequentially performing query linear transformation, key linear transformation and value linear transformation on the input data based on the query weight, key weight and value weight includes: Dividing the query weight, the key weight, and the value weight into sub-weights respectively; The sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight are distributed to each processing unit, and each processing unit is controlled to perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence based on the sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight.
[0049] Specifically, during the weight loading phase, the weight swizzle technique can be used to rearrange the data structure of query weights, key weights, and value weights so that these weights can match the memory access pattern of the processing unit, thereby achieving more efficient linear transformation.
[0050] First, the query weight, key weight, and value weight can be divided into multiple sub-weights. Each sub-weight corresponds to a part of the weight. By dividing the entire weight into smaller sub-weights, computing resources can be better managed and allocated. This division method enables the attention computing system to process large-scale weight matrices more flexibly, especially in high-dimensional data processing scenarios. The divided sub-weights are easier to distribute among multiple processing units, making full use of computing resources. For example, Figure 3 Schematic diagram of the weight shuffling technique in the fusion operator provided by the present invention. Figure 3 As shown in the figure, the query weight, key weight and value weight represented by Q, K and V can be divided into 16 sub-weights, which are shown as follows: Figure 3 "Q / 16", "K / 16", "V / 16" in Figure 3 The 16 sub-weights are divided as an example, and the weights may also be divided into 20, 30 or other numbers of sub-weights, which is not specifically limited in the embodiment of the present invention.
[0051] Next, the sub-weights of the divided query weight, key weight, and value weight can be respectively assigned to each processing unit. The processing unit here can be an independently operable computing unit on an artificial intelligence chip, such as a streaming processing cluster (SPC).
[0052] After receiving the sub-weights of the query weight, key weight, and value weight assigned to it, each processing unit is responsible for performing the corresponding query linear transformation, key linear transformation, and value linear transformation operations. Through this distributed processing method, linear transformation calculations can be performed in parallel on multiple processing units at the same time, avoiding the computational bottleneck caused by centralized processing in traditional methods.
[0053] Accordingly, in each processing unit, based on the sub-weights of the query weight, the sub-weights of the key weight, and the sub-weights of the value weight assigned to it, query linear transformation, key linear transformation, and value linear transformation can be performed on the input data in sequence. Specifically, each processing unit uses the sub-weights of the query weight assigned to it to convert the input data into a query vector; then uses the sub-weights of the key weight to convert the input data into a key vector; and finally uses the sub-weights of the value weight to convert the input data into a value vector. This process is performed independently and in parallel in each processing unit, greatly speeding up the overall calculation.
[0054] Based on the above embodiment, the number of the sub-weights is consistent with the number of the processing units.
[0055] Specifically, when partitioning query weights, key weights, and value weights, ensure that each weight matrix is partitioned into the same number of sub-weights as the number of processing units. This design choice means that each sub-weight in the weight matrix will be assigned to an independent processing unit to maximize the use of computing resources. For example, in Figure 3 There are 16 processing units, Spc0-Spc15, in total. When dividing the query weight, key weight and value weight, each weight is also divided into 16 sub-weights.
[0056] This consistency brings many advantages. First, since the number of sub-weights exactly matches the number of processing units, computing tasks can be allocated in an optimized manner, ensuring that each processing unit can receive a sub-weight and independently perform the computational operations of query linear transformation, key linear transformation, and value linear transformation. As a result, all processing units can perform calculations in the same time period, avoiding idle or uneven use of computing resources. Secondly, this design also simplifies the scheduling and management of the computing process. Since each processing unit only needs to process one sub-weight, no additional complex scheduling strategy is required when allocating tasks, and the sub-weights can be directly assigned to the processing units. This not only reduces the complexity of system scheduling, but also reduces the performance bottlenecks that may be caused by uneven task allocation.
[0057] Based on the above embodiment, when the query weight, key weight and value weight are quantized weights, the method further includes: Based on the fusion operator, loading scaling parameters; Before position encoding the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation, scaling the query vector and the key vector based on the scaling parameter; And, after obtaining the value vector obtained by the linear transformation of the value, scaling the value vector based on the scaling parameter.
[0058] During the operation of the fusion operator, a scaling operation is also introduced to further optimize the processing flow of query vectors, key vectors, and value vectors. Specifically, a set of scaling parameters can be loaded. These scaling parameters are tailored to the specific needs of the model and the characteristics of the input data, and are designed to adjust the numerical range of query vectors, key vectors, and value vectors to ensure that these vectors can maintain numerical stability in subsequent calculations and match the expected output range of the model.
[0059] After completing the query linear transformation and key linear transformation, the query vector and key vector can be obtained. Before position encoding these vectors, the query vector and key vector can be scaled based on the loaded scaling parameters. The scaling operation at this time is to adjust the numerical range of the query vector and key vector so that these vectors can maintain an appropriate numerical scale during the position encoding process, thereby avoiding calculation errors or numerical instability caused by too large or too small values. This scaling step is performed before position encoding to ensure that the position encoding can be performed under optimal numerical conditions, thereby improving the accuracy and stability of the encoding results.
[0060] In addition, after the linear transformation of the value is completed and the value vector is generated, the value vector will also be scaled based on the same scaling parameters. Similar to the scaling operation of the query vector and the key vector, the purpose of this step is to adjust the numerical range of the value vector so that it can be consistent with the scaling of the query vector and the key vector in the subsequent weighted summation process, thereby ensuring the accuracy of the final output result.
[0061] Based on the above embodiment, when the query weight, key weight and value weight are quantized weights, the method further includes: Based on the fusion operator, the scaled value vector is reconstructed.
[0062] Specifically, after scaling the value vector, the scaled value vector can also be reordered. The purpose of the reordering is to adjust the structure or content of the value vector according to the specific requirements of the model so that the reconstructed value vector can more effectively participate in the weighted summation process of the attention mechanism. The reordering process may include a series of data conversion, adjustment or optimization operations, which can be adjustments to the vector dimension, refinement of the vector content, or customized transformations for specific task requirements.
[0063] Based on the above embodiment, position encoding the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation includes: Load position encoding parameters; Based on the position encoding parameters, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0064] Specifically, before position encoding, a set of preset position encoding parameters can be loaded. These position encoding parameters are usually pre-set or learned from training data, and are used to embed the position information of the word in the input data into the query vector and key vector. The role of the position encoding parameters is to help the model understand the relative or absolute position of each word in the input data, so that the model can effectively take into account the position information when calculating the attention weight.
[0065] Next, based on the loaded position encoding parameters, the fusion operator positionally encodes the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation. The process of position encoding involves applying these parameters to the query vector and the key vector, usually through some mathematical transformation or function, such as sine and cosine functions, rotation position encoding, etc. Through this encoding, the query vector and the key vector not only contain the original feature information, but also incorporate the position information, so that they can reflect the relative position relationship of each word in the input data in the subsequent attention calculation.
[0066] Based on any of the above embodiments, Figure 4 Schematic diagram of the fusion operator calculation process provided by the present invention. Figure 4 As shown in the figure, during the operation of the fusion operator, the input data is transformed linearly through query, key, and value to obtain query, key, and value vectors. However, unlike traditional sequential processing, when performing the linear transformation of the value vector, the scaling operation and position encoding of the query and key vectors are already performed in parallel. This parallel computing method makes full use of computing resources and improves overall processing efficiency.
[0067] Specifically, during the operation of the fusion operator, when the input data is linearly transformed, the scaling and position encoding of the query vector and the key vector no longer wait for the completion of the linear transformation of the value vector, but are performed synchronously with it. This parallel processing method ensures that the query and key vectors can complete the position encoding and quantization as early as possible, thereby accelerating the subsequent attention calculation process. At the same time, after the linear transformation of the value is completed, the value vector is scaled and reconstructed, and quantized to further optimize the calculation efficiency. Through this design, the calculation tasks at different stages can be parallelized, reducing the overall calculation time and significantly improving the execution efficiency and resource utilization of the attention mechanism.
[0068] Based on any of the above embodiments, Figure 5 This is the second flow chart of the operation method of the attention mechanism provided by the present invention. Figure 5 As shown in the figure, first, the input data is sent to the query linear transformation, key linear transformation and value linear transformation modules respectively to generate query, key and value vectors. These linear transformations are completed through their respective weight matrices, and before that, these weight matrices have been processed by weight shuffling. The weight shuffling technology rearranges the weight matrices of query, key and value to optimize the memory access mode and computing efficiency, thereby providing a more efficient data organization method for subsequent calculations. In the process of performing these linear transformations, the fusion operator further improves the processing efficiency through parallel computing. In particular, when the value linear transformation module performs linear transformation of the value vector, the scaling operation and position encoding of the query and key vectors are already being performed synchronously. This parallel processing design can significantly reduce the computing waiting time and maximize the utilization of hardware resources.
[0069] The query and key vectors are scaled to adjust their numerical ranges to ensure numerical stability for subsequent calculations. At the same time, the value vector is also scaled after the linear transformation to adjust its numerical range.
[0070] Next, after scaling, the query and key vectors enter the position encoding and quantization stage to embed the position information into the vector so that the model can consider the position information of each word in the input data in the attention calculation. Before position encoding, the sine / cosine module generates sine and cosine values for position encoding, i.e., position encoding parameters, which are used to embed the position information into the query and key vectors. At the same time, after scaling, the value vector also undergoes reconstruction and quantization operations to optimize the data storage method and the efficiency of subsequent calculations.
[0071] The position-encoded query vector is output and stored in the corresponding cache, ready to be dot-producted with the key vector to calculate the attention weight. After position encoding and quantization, the key vector generates two results: key cache and key scaling. The key cache is used to store the main result of the key vector, while the key scaling is an additionally processed scaling result that may be used in specific computing scenarios or paths, providing more computing flexibility. After scaling, reconstructing, and quantizing, the value vector is stored in the value cache, waiting to be weighted summed in the attention mechanism. Similar to key scaling, value scaling may be used in specific computing scenarios, providing additional numerical adjustments and flexibility to ensure that the model can efficiently and stably process value vector data in various computing paths.
[0072] The operating device of the attention mechanism provided by the present invention is described below. The operating device of the attention mechanism described below and the operating method of the attention mechanism described above can be referenced to each other.
[0073] Figure 6 Schematic diagram of the structure of the operating device of the attention mechanism provided by the present invention. Figure 6 As shown, the device comprises: A fusion operator starting unit 610, used to start a fusion operator of an attention mechanism based on input data; The linear transformation and position encoding unit 620 is used to perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence based on the fusion operator, and while performing the value linear transformation, position encode the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0074] The device provided by the embodiment of the present invention integrates the independent calculation steps in the traditional attention mechanism into a unified operation by introducing a fusion operator, reduces the number of kernel startups, avoids frequent and repeated parameter loading, and thus greatly improves the calculation efficiency. In addition, during the operation of the fusion operator, position encoding is performed while the value is linearly transformed, realizing parallel processing of linear transformation and position encoding, avoiding the calculation bottleneck caused by sequential execution in traditional methods, and making the operation process of the attention mechanism more efficient.
[0075] Based on any of the above embodiments, the linear transformation and position encoding unit is specifically used for: Loading query weights, key weights and value weights required for performing the query linear transformation, key linear transformation and value linear transformation; Based on the query weight, key weight and value weight, query linear transformation, key linear transformation and value linear transformation are sequentially performed on the input data.
[0076] Based on any of the above embodiments, the linear transformation and position encoding unit is further used for: Dividing the query weight, the key weight, and the value weight into sub-weights respectively; The sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight are distributed to each processing unit, and each processing unit is controlled to perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence based on the sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight.
[0077] The number of the sub-weights is consistent with the number of the processing units.
[0078] Based on any of the above embodiments, the linear transformation and position encoding unit is further used for: Load position encoding parameters; Based on the position encoding parameters, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0079] Based on any of the above embodiments, the linear transformation and position encoding unit further includes: A scaling subunit is used to load scaling parameters based on the fusion operator when the query weight, key weight and value weight are quantized weights; before position encoding the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation, scale the query vector and the key vector based on the scaling parameters; and after obtaining the value vector obtained by the value linear transformation, scale the value vector based on the scaling parameters.
[0080] A reconstruction subunit is used to reconstruct the scaled value vector based on the fusion operator when the query weight, key weight and value weight are quantized weights.
[0081] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the operation method of the attention mechanism, which includes: Based on the input data, start the fusion operator of the attention mechanism; Based on the fusion operator, query linear transformation, key linear transformation and value linear transformation are performed on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0082] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0083] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the operation method of the attention mechanism provided by the above methods, the method comprising: Based on the input data, start the fusion operator of the attention mechanism; Based on the fusion operator, query linear transformation, key linear transformation and value linear transformation are performed on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0084] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the operation method of the attention mechanism provided by the above methods, the method comprising: Based on the input data, start the fusion operator of the attention mechanism; Based on the fusion operator, query linear transformation, key linear transformation and value linear transformation are performed on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for operating an attention mechanism, characterized in that: include: Based on the input data, start the fusion operator of the attention mechanism; Based on the fusion operator, query linear transformation, key linear transformation and value linear transformation are performed on the input data in sequence, and while performing the value linear transformation, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
2. The method for operating the attention mechanism according to claim 1, characterized in that: The step of sequentially performing a query linear transformation, a key linear transformation, and a value linear transformation on the input data includes: Loading query weights, key weights, and value weights required for performing the query linear transformation, key linear transformation, and value linear transformation; Based on the query weight, key weight and value weight, query linear transformation, key linear transformation and value linear transformation are sequentially performed on the input data.
3. The method for operating the attention mechanism according to claim 2, characterized in that: The step of sequentially performing query linear transformation, key linear transformation and value linear transformation on the input data based on the query weight, key weight and value weight includes: Dividing the query weight, the key weight, and the value weight into sub-weights respectively; The sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight are distributed to each processing unit, and each processing unit is controlled to perform query linear transformation, key linear transformation and value linear transformation on the input data in sequence based on the sub-weights of the query weight, the sub-weights of the key weight and the sub-weights of the value weight.
4. The method for operating the attention mechanism according to claim 3, characterized in that: The number of the sub-weights is consistent with the number of the processing units.
5. The method for operating the attention mechanism according to claim 2, characterized in that: In the case where the query weight, key weight and value weight are quantized weights, the method further includes: Based on the fusion operator, loading scaling parameters; Before position encoding the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation, scaling the query vector and the key vector based on the scaling parameter; And, after obtaining the value vector obtained by the linear transformation of the value, scaling the value vector based on the scaling parameter.
6. The method for operating the attention mechanism according to claim 5, characterized in that: In the case where the query weight, key weight and value weight are quantized weights, the method further includes: Based on the fusion operator, the scaled value vector is reconstructed.
7. The method for operating the attention mechanism according to claim 1, characterized in that: The performing position encoding on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation comprises: Load position encoding parameters; Based on the position encoding parameters, position encoding is performed on the query vector obtained by the query linear transformation and the key vector obtained by the key linear transformation.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the operating method of the attention mechanism as described in any one of claims 1 to 7.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the operation method of the attention mechanism as described in any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the operation method of the attention mechanism as described in any one of claims 1 to 7.
Citation Information
Cited By
Heterogeneous system and method for efficiently realizing attention mechanism of large language model and application
CN121436044A
Visual task-oriented linear attention mechanism method and device and electronic equipment
CN122047306A
Linear attention mechanism methods, devices and electronic devices for vision tasks
CN122047306B
