A rotating position encoding acceleration method, device, equipment and medium

Abstract

The application discloses a rotation position encoding acceleration method, device, equipment and medium. The method comprises the following steps: obtaining at least one word piece block, and carrying the attention head data of each word piece in the target word piece block and the rotation coefficient vector to the first-level cache; determining a multi-round loop calculation strategy, and broadcasting the rotation coefficient vector of each word piece; determining the instruction channel number of the current calculation round according to the multi-round loop calculation strategy; if the instruction channel number is greater than 1, the attention head data is read in a stride reading mode, the first rotation component and the second rotation component of a plurality of attention head vectors in each instruction channel are obtained, and the first output component or the second output component of each attention head vector is calculated in each instruction channel; a splicing vector is generated, and the splicing vector is written into the global memory. By using the above technical scheme, the utilization rate of the vector register can be improved, the memory access delay can be reduced, and the efficient execution of the rotation position encoding is realized.

Description

Technical Field

[0001] This invention relates to the field of model reasoning acceleration technology, and in particular to a method, apparatus, device and medium for accelerating rotational position encoding. Background Technology

[0002] Rotary Position Embedding (RoPE) is the most widely used positional encoding scheme in the attention mechanism of current large language models. Its core design idea is to embed absolute positional information into the query and key header vectors of attention computation through rotation transformation in the complex field. There are two main data arrangement styles for RoPE: one is the NeoX style, in which the first half of the attention head stores the x-component and the second half stores the y-component; the other is the non-NeoX style, in which even-numbered indices store the x-component and odd-numbered indices store the y-component.

[0003] Existing rotation position encoding implementations are mostly general-purpose implementations. Under the SIMD (Single Instruction Multiple Data) architecture, a single SIMD instruction processes only the data of one attention head. When reading the rotation coefficient vectors cos and sin vectors, they are read one by one from the global storage in a word-by-word and attention-head-by-attention manner. When writing back the result, a step-by-step storage operation is performed on the updated x and y components respectively.

[0004] However, the number of elements that a single SIMD instruction can process far exceeds the half-rotation dimension of a single attention head. If a SIMD instruction only processes the data of one attention head, the utilization rate of the vector register will be extremely low, resulting in a significant waste of computing power. The method of moving the rotation coefficient vector by word and attention head will lead to a large number of irregular memory accesses and duplicate data movements, which will severely restrict the efficiency of the computing pipeline. When writing back the result, performing step-by-step storage operations on the x and y components will bring significant memory access latency and instruction overhead. The rotation position encoding schemes in the existing technology have extremely low encoding efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for accelerating rotational position encoding, which can improve vector register utilization, reduce memory access latency, and achieve efficient execution of rotational position encoding.

[0006] According to one aspect of the present invention, a method for accelerating rotational position encoding is provided, comprising: The tokens allocated to the current thread are segmented to obtain at least one token block. When the processing of the target token block is started, the attention header data and rotation coefficient vector of each token in the target token block are moved to the first-level cache. Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, a multi-round loop computation strategy is determined, and the rotation coefficient vectors of each word are broadcast in the first-level cache. When performing rotational position encoding on the first type of attention head vector of the target word, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, the attention head data is read from the first-level cache using the step-read method, and the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained. In each instruction channel, the first output component or the second output component of each attention head vector is calculated based on the first rotation component, the second rotation component and the rotation coefficient vector after broadcast. Based on the first and second output components of the attention head vector, a concatenated vector is generated and written into global memory to interleave the elements of the first and second output components in global memory.

[0007] According to another aspect of the present invention, a rotational position encoding acceleration device is provided, comprising: The data transport module is used to segment the lexical units allocated to the current thread, obtain at least one lexical unit block, and when the processing of the target lexical unit block is started, transport the attention head data and rotation coefficient vector of each lexical unit in the target lexical unit block to the first-level cache. The loop calculation strategy determination module is used to determine the multi-round loop calculation strategy based on the maximum number of vectors that a single calculation instruction can process and the size of the attention head of a single word, and broadcast the rotation coefficient vector of each word in the first-level cache. The calculation round iteration module is used to determine the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy when performing rotation position encoding on the first type of attention head vector of the target word. The multi-instruction channel calculation module is used to read the attention head data in the first-level cache by step reading if the number of instruction channels is greater than 1, obtain the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel, and calculate the first output component or the second output component of each attention head vector in each instruction channel according to the first rotation component, the second rotation component and the rotation coefficient vector after broadcast. The merge-write-back module is used to generate a concatenated vector based on the first and second output components of the attention head vector, and write the concatenated vector into global memory so that the elements in the first and second output components are interleaved in global memory.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the rotation position encoding acceleration method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the rotation position encoding acceleration method according to any embodiment of the present invention.

[0010] The technical solution of this invention involves segmenting the lexical units allocated to the current thread to obtain at least one lexical unit block. When processing a target lexical unit block is initiated, the attention head data and rotation coefficient vectors of each lexical unit in the target lexical unit block are moved to the first-level cache. A multi-round cyclic calculation strategy is determined based on the maximum number of vectors that a single calculation instruction can process and the size of the attention head of a single lexical unit, and the rotation coefficient vectors of each lexical unit are broadcast in the first-level cache. When encoding the rotation position of the first type of attention head vector for the target lexical unit, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, a step-by-step reading method is used to read the number of attention heads from the first-level cache. According to the method, the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained, and the first output component or the second output component of each attention head vector is calculated in each instruction channel. Based on the first output component and the second output component, a concatenated vector is generated and written into global memory. By interleaving the elements in the first output component and the second output component in global memory, the utilization of vector registers can be greatly improved, computational waste can be reduced, the memory access overhead of rotation coefficient vector can be reduced, duplicate data handling can be avoided, the number of write-back stage storage instructions can be effectively reduced, the overall computational efficiency can be improved, and the pipeline of data handling and computation can be overlapped to maximize the utilization of hardware resources.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a rotation position encoding acceleration method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another rotation position encoding acceleration method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a rotational position encoding acceleration device according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the rotation position encoding acceleration method of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a rotational position encoding acceleration method provided in Embodiment 1 of the present invention. This embodiment is applicable to the efficient rotational position encoding of non-NeoX style attention head vectors. The method can be executed by a rotational position encoding acceleration device, which can be implemented in hardware and / or software, and is generally configured in a computer or processor with data processing capabilities. Figure 1 As shown, the method includes: S110. Segment the tokens allocated to the current thread to obtain at least one token block. When the processing of the target token block is started, move the attention header data and rotation coefficient vector of each token in the target token block to the first-level cache.

[0017] Optionally, the present invention can be applied to AI (Artificial Intelligence) acceleration chips under the SIMD architecture. SIMD refers to the ability of a single operation instruction to perform the same operation on multiple independent data in parallel. The SIMD architecture is a hardware parallel architecture for AI acceleration chips that is oriented towards massive vector operations.

[0018] Optionally, a word is the basic unit of text processing in a large language model. It can be understood as a single element obtained after segmenting the input text of the large language model.

[0019] Optionally, lexical segmentation refers to dividing all the lexical units that the current thread is responsible for processing into subsets based on the capacity limit of the first-level cache, ensuring that all data in each subset can be placed into the first-level cache; the target lexical segmentation is the lexical segmentation currently being processed.

[0020] Optionally, the first-level cache, also known as the L1 cache, is the high-speed cache in the AI ​​acceleration chip that is closest to the computing unit and has the lowest access latency.

[0021] Optionally, the attention head data may include multiple attention head vectors. The attention head vector is a one-dimensional vector and is divided into two types: query head vector and key head vector. The query head vector is also known as the query vector, and the key head vector is also known as the key vector. In the attention head data of a word, the number of query head vectors and key head vectors is the same.

[0022] Optionally, each attention head vector consists of a first rotation component and a second rotation component. The first rotation component is also known as the x component, which is the real part of the attention head vector that participates in the rotation transformation. The second rotation component is also known as the y component, which is the imaginary part of the attention head vector that participates in the rotation transformation.

[0023] Optionally, if the attention head vector is non-NeoX style, the first rotation component and the second rotation component are interleaved. That is, within an attention head vector, the even-numbered index positions store the first rotation component, and the odd-numbered index positions store the second rotation component. For example, a non-NeoX style attention head vector can be represented as: (x0, y0, x1, y1, ..., x n-1 y n-1 ), where x0, x1, ..., x n-1 Together they form the x-component, also known as the first rotational component, y0, y1, ..., y2. n-1 Together they form the y-component, which is also the second rotational component.

[0024] Optionally, the rotation coefficient vector is the coefficient vector used when rotating the attention head vector for position encoding. The rotation coefficient vector can be pre-computed and stored in global memory. Each word corresponds to a set of independent rotation coefficient vectors. The rotation coefficient vector can include a first rotation coefficient vector and a second rotation coefficient vector. The first rotation coefficient vector can be understood as an array of cos values ​​in rotation position encoding, and the second rotation coefficient vector can be understood as an array of sin values ​​in rotation position encoding.

[0025] Optionally, the maximum number of target words that can be stored in the first-level cache can be determined based on the total capacity of the first-level cache and the total amount of data for a single word word. The total amount of data for a single word word can be determined based on the total number of query header vectors, the total number of key header vectors, the rotation coefficient vector, and the length of a single vector.

[0026] Furthermore, if the total number of tokens allocated by the current thread is less than or equal to the target number of tokens, it is treated as a single block; otherwise, all tokens allocated by the current thread are divided into multiple blocks based on the target number of tokens.

[0027] Optionally, when processing each word segment, all the first rotation coefficient vectors, second rotation coefficient vectors, and rotation coefficient vectors of that word segment are moved from global memory to the corresponding area of ​​the first-level cache.

[0028] Optionally, global memory is a large-capacity off-chip storage space for AI acceleration chips. Its access latency is much higher than that of the L1 cache. Global memory is used for long-term storage and management of attention head vectors, rotation coefficient vectors, and rotation position encoding results. Data in global memory is only moved to the L1 cache when AI acceleration chips need to perform calculations.

[0029] Before segmenting the tokens allocated to the current thread and obtaining at least one token block, the process may further include: Based on the vector register width and the target data type used in the rotation position encoding, determine the maximum number of elements that a single computation instruction can process in parallel, and determine the half-rotation dimension based on the rotation dimension of the rotation position encoding. Based on the maximum number of elements and the semi-rotation dimension, determine the maximum number of vectors that a single computation instruction can process.

[0030] Optionally, the vector register width is a fixed hardware parameter of the AI ​​acceleration chip. In AI acceleration chips under the SIMD architecture, large-width vector registers are often used, and the vector register width can be 512 bytes, 1024 bytes, etc.

[0031] Optionally, the target data type refers to the data type used in the rotation position encoding calculation, which is mostly a 16-bit floating-point number. In high-precision calculation scenarios, a 32-bit floating-point number can also be used. There is no limitation here.

[0032] Optionally, the maximum number of elements refers to the total number of elements that a single SIMD vector instruction can process in parallel. Based on the target data type, the number of bytes occupied by a single element can be determined, and then the quotient between the vector register width and the number of bytes occupied by a single element can be calculated. The result is the maximum number of elements that a single computation instruction can process in parallel.

[0033] In an optional example, if the vector register width is 512 bytes and the target data type is a 16-bit floating-point number, then a single element occupies 2 bytes, and the maximum number of elements that a single computation instruction can process in parallel is 512 ÷ 2 = 256. That is, a single SIMD vector instruction can process 256 elements in parallel at once.

[0034] Optionally, the rotation dimension of rotational position encoding refers to the number of elements involved in the rotational position encoding calculation in each attention head vector. The rotation dimension is the total number of elements in the first rotation component and the second rotation component in a single attention head; the half-rotation dimension is half of the rotation dimension, that is, the number of elements in the first rotation component or the second rotation component in a single attention head; continuing the previous example, if an attention head vector is represented as (x0, y0, x1, y1, ..., x... n-1 y n-1 If the rotation dimension is n, then the half-rotation dimension is n / 2. The rotation dimension can be obtained directly by reading the configuration file of the large language model.

[0035] Understandably, in large language models, typical rotation dimensions are generally 128 or 64, while the corresponding half-rotation dimensions are 64 or 32. If a vector register with a width of 512 bytes is used, a single computation instruction can process 256 elements in parallel. If each computation instruction processes only one attention head vector, the utilization rate of the vector register is only 64÷256=25%, or 32÷256=12.5%. It can be seen that the utilization rate of the vector register is extremely low, which will cause a large waste of computing power. Therefore, this invention creatively proposes to process multiple attention head vectors simultaneously in a single computation instruction, thereby improving the utilization rate of the vector register.

[0036] Optionally, the maximum number of vectors that a single computation instruction can process refers to the number of attention head vectors that a single SIMD instruction can process simultaneously. The quotient of the maximum number of elements and the half-rotation dimension is the maximum number of vectors. If the maximum number of elements is identified by vlen, the half-rotation dimension is represented by half_rot, and the maximum number of vectors that a single computation instruction can process is represented by ratio, then ratio = vlen / half_rot.

[0037] Optionally, the maximum number of vectors that a single computation instruction can process is a positive integer. If there is a non-standard model configuration that results in the inability to divide, the integer part is rounded down to ensure that the elements of each attention head are continuously aligned in the vector register, thus avoiding the additional overhead caused by cross-register access.

[0038] The advantages of this setup are: based on the rotation dimension configured in the large language model and the vector register width provided by the AI ​​acceleration chip, multiple attention head vectors can be processed simultaneously in a single SIMD instruction, which can improve the utilization rate of the vector register to nearly 100%. The method for calculating the maximum number of vectors can be adapted to various combinations of rotation dimensions and data types, effectively avoiding waste of computing power.

[0039] S120. Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, determine the multi-round loop computation strategy and broadcast the rotation coefficient vector of each word in the first-level cache.

[0040] Optionally, the attention head size of a single word refers to the total number of attention heads corresponding to a single word. Based on the attention head size of a single word, the number of query head vectors and key head vectors corresponding to a single word can be determined.

[0041] Optionally, the multi-round loop calculation strategy refers to a multi-layered loop execution scheme based on the single-instruction parallel processing capability for the query head vector or key head vector of a single word. This includes a main loop for full instruction channel calculation, an auxiliary loop for integer multiple residual instruction channel calculation, and a tail loop for calculations that do not reach the maximum number of vectors.

[0042] Optionally, a 16-way loop expansion can be used to perform rotation position encoding calculations on the attention head vectors. If the maximum number of vectors that a single calculation instruction can process is represented by the ratio, then a single loop calculation can simultaneously calculate up to ratio × 16 attention head vectors.

[0043] Optionally, since a single computation instruction processes ratio attention head vectors simultaneously, and the original width of the rotation coefficient vector prefetched to the L1 cache temporary region is equal to half the rotation dimension half_rot, the rotation coefficient vector cannot match the processing width of a single SIMD instruction. Furthermore, all attention head vectors in the same term share the same rotation coefficient vector. Therefore, the data transport engine channel can be called to broadcast ratio copies of the rotation coefficient vector to ensure data alignment during computation.

[0044] S130. When performing rotational position encoding on the first type of attention head vector of the target word, determine the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy.

[0045] Optionally, the target word is the word being processed. The first type of attention head vector can refer to the attention head vector of a specified type being processed, which can be a query head vector or a key head vector. During the rotation and position encoding of the attention head vector, the attention head vector of a single word can be divided into two batches for processing according to its type.

[0046] Optionally, an instruction channel refers to an independent instruction execution stream, with each instruction channel corresponding to an independent vector register.

[0047] Optionally, if a 16-way cyclic expansion is used, it means that 16 different attention head data are processed simultaneously through 16 instruction channels. The maximum number of instruction channels is 16 and the minimum is 1.

[0048] Optionally, in a multi-round cyclic computation strategy, the number of instruction channels used in each round of cyclic computation can be clearly defined; if the number of instruction channels is greater than 1, attention head vectors that meet the maximum number of vectors are processed in each instruction channel; if the number of instruction channels is equal to 1, only 1 attention head vector is processed in that instruction channel.

[0049] The advantage of this setup is that each instruction channel is completely independent, allowing for overlapping execution within the hardware pipeline, thus masking memory access latency and instruction pipeline pauses.

[0050] S140. If the number of instruction channels is greater than 1, the attention head data is read from the first-level cache using the step-read method, and the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained. In each instruction channel, the first output component or the second output component of each attention head vector is calculated according to the first rotation component, the second rotation component and the rotation coefficient vector after broadcast.

[0051] Optionally, if the number of instruction channels is greater than 1, the number of attention head vectors in the instruction channel is equal to the maximum number of vectors that a single computation instruction can process.

[0052] Understandably, in non-NeoX style attention head vectors, the first rotation component and the second rotation component are interleaved, which can be represented as: (x0, y0, x1, y1, ..., x n-1 y n-1 During the calculation, the x-component and y-component need to be obtained separately. Therefore, the step-read method is required to read the x-component and y-component of all attention head vectors in each instruction channel.

[0053] Optionally, since the first rotation component is stored at an even-numbered index and the second rotation component is stored at an odd-numbered index, a step-load instruction with a step size of 2 can be executed starting from an even-numbered address to read all the first rotation components spanning the maximum number of attention head vectors ratio at once. Then, starting from an odd-numbered address, a step-load instruction with a step size of 2 can be executed to read all the second rotation components spanning the maximum number of attention head vectors ratio at once.

[0054] Furthermore, after completing the reading of the first and second rotation components of an instruction channel, the starting address is advanced by ratio×rot_dim elements, which is the product of the maximum number of vectors and the rotation dimension, so that the processed attention head vector is skipped in the next reading.

[0055] Optionally, after reading the first rotation component and the second rotation component in the current instruction channel, the read information and the broadcast first rotation coefficient vector and the second rotation coefficient vector are used to perform rotation position encoding calculation in the current instruction channel.

[0056] Optional, can be based on the formula Calculate the first output component according to the formula. Calculate the second output component; Where x_new is the first output component, y_new is the second output component; cos represents the cosine value array, which is also the first rotation coefficient vector; sin represents the sinine value array, which is also the second rotation coefficient vector; x is the first rotation component; y is the second rotation component.

[0057] S150. Generate a concatenated vector based on the first and second output components of the attention head vector, and write the concatenated vector into global memory so that the elements of the first and second output components are interleaved in global memory.

[0058] Understandably, after calculating the first and second output components, they need to be written back to global memory in their original interleaved arrangement, for example, in the following form: (x_new0, y_new0, x_new1, y_new1, ..., x_new0, y_new1, ..., x_new1, y_new2, y_new3, ..., x_new4, y_new5, y_new6, y_new7, y_new8, y_new9, y_new1, ..., x_new1, y_new2, y_new3, y_new4, y_new5 ...1, ..., x_new1, y_new2, n-1 y_new n-1 In the prior art, a straddle memory operation needs to be performed on the first output component and the second output component respectively. The two straddle memory operations result in double the memory instruction overhead and pipeline occupation, which will lead to significant memory access latency.

[0059] Optionally, the concatenated vector is a vector generated by merging the first and second output components through bitwise operations. Specifically, the corresponding elements in the first and second output components are concatenated, and the concatenated elements are arranged according to a specified positional relationship to generate the concatenated vector. The concatenated vector is written into global memory, which can directly meet the requirements of the staggered arrangement.

[0060] Optionally, each element of the concatenated vector contains both an element from the first output component and an element from the second output component. Continuing the previous example, the concatenated vector can be represented as (out0, out1, ..., out...). n-1 ), where out0 can represent the element obtained by concatenating x_new0 and y_new0, and out1 can represent the element obtained by concatenating x_new1 and y_new1, and so on.

[0061] Optionally, by generating a splicing vector, two straddle storages can be merged into a single normal storage, and the cross storage of the first and second output components can be performed with a single storage instruction.

[0062] The process of generating the concatenated vector based on the first and second output components of the attention head vector can include: When the data type of the first output component and the second output component is 16-bit floating-point number, each element in the first output component and the second output component is converted into a 16-bit unsigned integer, and after the conversion, each element in the first output component and the second output component is extended from a 16-bit unsigned integer to a 32-bit unsigned integer. After shifting each expanded element in the second output component 16 bits to the left, perform a bitwise OR operation with each expanded element in the first output component to generate a concatenated vector.

[0063] Optionally, each element in the first and second output components can be converted from a 16-bit floating-point number to a 16-bit unsigned integer while maintaining the binary bit pattern.

[0064] Optionally, for each element converted to a 16-bit unsigned integer, its high 16 bits are padded with 0 to generate a 32-bit unsigned integer. In the extended 32-bit unsigned integer, the high 16 bits are 0 and the low 16 bits are the original 16-bit unsigned integer value of the element.

[0065] Optionally, the expanded elements in the second output component are shifted left by 16 bits. At this time, the high 16 bits of each element in the second output component are the original value of the 16-bit unsigned integer of that element, and the low 16 bits are 0. Then, a bitwise OR operation is performed between each element in the left-shifted second output component and each expanded element in the first output component to obtain a 32-bit concatenated vector element.

[0066] Optionally, a bitwise OR operation means that the result bit is 1 if one of the two corresponding bits is 1, and 0 otherwise.

[0067] Optionally, the lower 16 bits of each element in the concatenated vector are elements in the first output component, and the higher 16 bits are elements in the second output component.

[0068] In an optional example, if the 16-bit unsigned integer converted from x_new1 in the first output component is 00010010 0011 0100, then it is converted to a 32-bit unsigned integer as 0000 0000 0000 0000 0001 0010 00110100; if the 16-bit unsigned integer converted from y_new1 in the second output component is 0101 0110 0111 1000, then it is converted to a 32-bit unsigned integer as 0000 0000 0000 0000 0101 0110 0111 1000. Shifting y_new1 left by 16 bits gives 0101 0110 0111 1000 0000 0000 0000 0000; After performing a bitwise OR operation between the left-shifted y_new1 and the 32-bit x_new1, we get 0101 0110 0111 1000 0001 0010 0011 0100. At this point, we can see that the high 16 bits are y_new1 and the low 16 bits are x_new1.

[0069] It is understandable that when performing normal storage, writing 32 bits of data at once will first write the lower 16 bits and then the higher 16 bits. Therefore, according to the write-back order of normal storage, when writing back each element in the concatenated vector, the elements of the first output component are written back first, and then the elements of the second output component are written back, which corresponds to the alternating arrangement of the first and second output components.

[0070] The advantages of this setup are: it halves the number of memory instructions for write-back operations, eliminates the address calculation overhead of straddle memory, and significantly reduces the number of memory access instructions and pipeline stalls.

[0071] The technical solution of this invention involves segmenting the lexical units allocated to the current thread to obtain at least one lexical unit block. When processing a target lexical unit block is initiated, the attention head data and rotation coefficient vectors of each lexical unit in the target lexical unit block are moved to the first-level cache. A multi-round cyclic calculation strategy is determined based on the maximum number of vectors that a single calculation instruction can process and the size of the attention head of a single lexical unit, and the rotation coefficient vectors of each lexical unit are broadcast in the first-level cache. When encoding the rotation position of the first type of attention head vector for the target lexical unit, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, a step-by-step reading method is used to read the number of attention heads from the first-level cache. According to the method, the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained, and the first output component or the second output component of each attention head vector is calculated in each instruction channel. Based on the first output component and the second output component, a concatenated vector is generated and written into global memory. By interleaving the elements in the first output component and the second output component in global memory, the utilization of vector registers can be greatly improved, computational waste can be reduced, the memory access overhead of rotation coefficient vector can be reduced, duplicate data handling can be avoided, the number of write-back stage storage instructions can be effectively reduced, the overall computational efficiency can be improved, and the pipeline of data handling and computation can be overlapped to maximize the utilization of hardware resources.

[0072] Example 2 Figure 2 This is a flowchart of a rotational position encoding acceleration method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifically describes the rotational position encoding acceleration method. Figure 2 As shown, the method includes: S210. Based on the vector register width and the target data type used in the rotation position encoding, determine the maximum number of elements that a single computation instruction can process in parallel, and determine the half-rotation dimension based on the rotation dimension of the rotation position encoding.

[0073] S220. Determine the maximum number of vectors that a single computation instruction can process based on the maximum number of elements and the semi-rotation dimension.

[0074] S230. Segment the tokens allocated to the current thread to obtain at least one token block. When the processing of the target token block is started, move the attention header data and rotation coefficient vector of each token in the target token block to the first-level cache.

[0075] The process of segmenting the tokens allocated to the current thread to obtain at least one token block may include: Based on the maximum number of vectors that a single computation instruction can process, the size of the attention head of a single word, and the capacity of the first-level cache, multiple storage areas are divided in the first-level cache; Based on the storage area partitioning results and the rotation dimension of the rotation position encoding, determine the number of target words and determine whether the number of target words is greater than or equal to the total number of words allocated to the current thread. If so, the tokens allocated to the current thread will be divided into a token block; If not, then based on the number of target lexical units, the lexical units allocated to the current thread are segmented to obtain multiple lexical unit blocks.

[0076] Optionally, the capacity of the first-level cache refers to the total capacity of the first-level cache that the current thread can exclusively use.

[0077] Optionally, the first-level cache can be pre-divided into a temporary storage area, a first rotation coefficient vector storage area, a second rotation coefficient vector storage area, a query header vector storage area, and a key header vector storage area.

[0078] Optionally, the temporary storage area is used to store the first rotation coefficient vector and the second rotation coefficient vector corresponding to all words in the word block; the first rotation coefficient vector storage area is used to store the first rotation coefficient vector after broadcast; the second rotation coefficient vector storage area is used to store the second rotation coefficient vector after broadcast; the query header vector storage area is used to store the query header vector of all words in the word block; and the key header vector storage area is used to store the key header vector of all words in the word block.

[0079] Optionally, if ratio represents the maximum number of vectors that a single computation instruction can process, and m represents the number of query head vectors or key head vectors corresponding to a single token, then the capacity ratio of the temporary storage area, the first rotation coefficient vector storage area, the second rotation coefficient vector storage area, the query head vector storage area, and the key head vector storage area in the first-level cache can be 1:ratio / 2:ratio / 2:m:m.

[0080] Optionally, based on the storage area partitioning results, the number of elements that the temporary storage area can store can be determined, and then the quotient between the number of elements that the temporary storage area can store and the half-rotation dimension can be calculated. The result is rounded down to obtain the number of target word elements.

[0081] The attention header data includes multiple query header vectors and multiple key header vectors.

[0082] Optionally, the query head vector is used to describe the features of a single word, and the key head vector is used to describe the index features of each dimension in the word sequence. The attention score can be obtained by calculating the inner product of the query head vector and the key head vector, thereby identifying the strength of semantic association between words. By performing rotational position encoding on the query head vector and the key head vector, the disorder defect of large language models can be solved, so that the attention score can reflect the relative positional relationship between words.

[0083] Moving the attention header data and rotation coefficient vectors of each word in the target word block to the first-level cache may include: Through two data transfer engine channels, the query header vector and key header vector of each word in the target word block are transferred from global memory to the query header vector storage area and key header vector storage area in the first-level cache, respectively. Through multiple rounds of iterative loops, the rotation coefficient vectors of each word within the target word block are moved from global memory to a temporary storage area in the first-level cache in batches. In each round of iterative loop, the rotation coefficient vectors of the specified word are moved by multiple parallel loading instructions. The rotation coefficient vectors include a first rotation coefficient vector and a second rotation coefficient vector.

[0084] Optionally, the Data Transfer Engine (DTE) is a dedicated hardware data transfer unit in the AI ​​acceleration chip. It can perform data transfer independently of the computing unit. Data transfer does not consume vector operation computing power and can achieve overlap between memory access and computing pipelines.

[0085] Optionally, the dual-channel data transport engine asynchronously transports the query header vector and the key header vector respectively. During the transport of the attention header vector, the data prefetching of the rotation coefficient vector can be performed in parallel to achieve the overlap of calculation and transport.

[0086] Optionally, in the rotation position encoding calculation, all attention heads of each word share the same rotation coefficient vector. However, the reading position of the rotation coefficient vector is determined by the position index, which points to a specific row in global memory, forming an indirect addressing mode. Moreover, the position index can only be determined at runtime, resulting in unpredictable memory access patterns when reading the rotation coefficient vector. Therefore, the rotation coefficient vector cannot be directly moved by the data moving engine. Instead, a vector loading instruction is needed to load the rotation coefficient vector from the corresponding position in global memory into the temporary storage area in the first-level cache.

[0087] Optionally, a multi-way loop expansion technique can be used to realize the transfer of rotation coefficient vectors through multiple rounds of loop iteration. The multi-way loop expansion can be a 16-way loop expansion, which is not limited here.

[0088] Optionally, parallel loading instructions can refer to multiple independent vector loading instructions. Parallel loading instructions can be executed overlappingly in the pipeline to make full use of memory bandwidth.

[0089] Optionally, through multiple rounds of iterative loops, the rotation coefficient vectors of each word within the target word block are moved from global memory to a temporary storage area in the first-level cache in batches. This can specifically include: In each iteration, the position indices of multiple words are read into the position register array, and the rotation coefficient vector address of each word is calculated based on the position indices in the position register array. The parallel loading instruction is invoked to perform the operation of loading each rotation coefficient vector into the coefficient register array in parallel, based on the address of the rotation coefficient vector of each word and the rotation dimension. Batch call to store instructions writes the rotation coefficient vectors stored in the coefficient register array to the temporary storage area in the L1 cache.

[0090] Optionally, taking a 16-way loop expansion as an example, each loop iteration reads the position index values ​​of 16 words into the position register array pos

[16] . Based on the position indexes in the register array pos

[16] and the starting address of the rotation coefficient vector in global memory, the address of the rotation coefficient vector of each word can be determined. Parallel loading instructions are called, and each loading instruction loads the rotation coefficient vector into the coefficient register array cos_sin

[16] according to the rotation coefficient vector address and rotation dimension of the specified word. Batch storage instructions are called to write the rotation coefficient vector in the coefficient register array cos_sin

[16] into the temporary storage area in the first-level cache.

[0091] Optionally, multi-way loop unrolling allows load instructions to be executed overlappingly in the pipeline, effectively hiding the latency of a single indirect addressing.

[0092] S240. Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, determine a multi-round loop computation strategy and broadcast the rotation coefficient vector of each word in the first-level cache.

[0093] Optionally, when broadcasting the rotation coefficient vector, the data transfer engine can be invoked to retrieve the first rotation coefficient vector and the second rotation coefficient vector from the temporary storage area in the first-level cache, and broadcast the first rotation coefficient vector to the first rotation coefficient vector storage area in the first-level cache, and broadcast the second rotation coefficient vector to the second rotation coefficient vector storage area in the first-level cache. The number of broadcasts is equal to the maximum number of vectors that a single computation instruction can process.

[0094] Optionally, after broadcasting the rotation coefficients, the number of elements in the first rotation coefficient vector and the second rotation coefficient vector corresponding to each lexical can be equal to the maximum number of elements that a single computation instruction can process in parallel, matching the processing width of the SIMD instruction.

[0095] The advantage of this setup is that it collaboratively achieves zero-wait data provisioning by prefetching the rotation coefficient vector into the L1 cache, masking memory access latency using multi-way loop unrolling, and broadcasting the rotation coefficient vector to match the processing width of SIMD instructions.

[0096] Among them, the multi-round iterative computation strategy is determined based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, which may include: Based on the maximum number of vectors that a single instruction can process and the preset maximum number of instruction channels, determine the first number of vectors that can be calculated using all instruction channels in a single round; Based on the attention head size of a single word and the number of the first vectors, determine at least one full instruction channel computation round and a first remaining vector that cannot be computed by the full instruction channel computation rounds; If the first remaining vector is greater than the maximum number of vectors that a single instruction can process, then based on the maximum number of vectors that a single instruction can process and the first remaining vector, the target number of computing channels in the multi-instruction channel computing round is determined, and the second remaining vector that cannot be computed in the multi-instruction channel computing round is determined. Based on the second remaining vector, at least one single-instruction single-vector calculation round is determined.

[0097] Optionally, the first vector number refers to the total number of attention head vectors that can be processed in a single round of full instruction channel calculation. The first vector number can be the product of the preset maximum number of instruction channels and the maximum number of vectors that a single instruction can process. For example, if a 16-way loop expansion is used, the maximum number of instruction channels is 16, and the maximum number of vectors that a single instruction can process is represented as ratio. Then, the result of 16 × ratio is the first vector number.

[0098] Optionally, the full instruction channel computation rounds can be used as the main loop for rotation position encoding. In each iteration of the main loop, all instruction channels are enabled simultaneously, and the number of attention head vectors processed by each instruction channel meets the maximum number of vectors.

[0099] Optionally, the first remaining vector can refer to the unprocessed attention head vector remaining after all computation rounds of the entire instruction channel have been executed.

[0100] Optionally, the multi-instruction channel computation rounds can be used as an auxiliary loop for rotation position encoding. Each iteration of the auxiliary loop can enable multiple instruction channels, but the number of instruction channels enabled at the same time is less than the maximum number of instruction channels. The number of attention head vectors processed by each instruction channel satisfies the maximum number of vectors and is used to process the integer multiple of the maximum number of vectors in the first remaining vector.

[0101] Optionally, the target number of channels is the incomplete quotient of the number of vectors in the first remaining vector and the maximum number of vectors that a single instruction can process.

[0102] Optionally, the product of the target number of channels and the maximum number of vectors that a single instruction can process is the total number of vectors that can be processed in a multi-instruction channel calculation round.

[0103] Optionally, the second remaining vector refers to the unprocessed attention head vector remaining after the multi-instruction channel auxiliary loop processing is completed.

[0104] In an optional example, if the total number of query header vectors or key header vectors for a word element is 300, the maximum number of vectors that a single instruction can process is 8, and the preset maximum number of instruction channels is 16, then the first number of vectors that can be calculated using all instruction channels in a single round is 16 × 8 = 128. A maximum of two rounds of calculation using all instruction channels can be executed. A total of 256 vectors can be processed through two rounds of calculation using all instruction channels. 300 - 256 = 44, that is, the remaining 44 vectors are the first remaining vectors. After processing them through 5 instruction channels at the maximum number of vectors, 40 vectors can be processed. The remaining 4 vectors are not processed. That is, the target number of calculation channels is 5. Finally, the remaining 4 vectors are the second remaining vectors.

[0105] Optionally, the single-instruction single-vector computation round can be used as the tail loop of the rotation position encoding. Each iteration of the tail loop only uses one instruction channel, and only one attention head vector is processed in the instruction channel to process the second residual vector one by one. Continuing the previous example, if there are four second residual vectors, they are processed through four single-instruction single-vector computation rounds.

[0106] S250. When performing rotational position encoding on the first type of attention head vector of the target word, determine the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy.

[0107] S260. If the number of instruction channels is greater than 1, the attention head data is read from the first-level cache using the step-read method, and the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained. In each instruction channel, the first output component or the second output component of each attention head vector is calculated based on the first rotation component, the second rotation component and the rotation coefficient vector after broadcast.

[0108] Specifically, when performing rotational position encoding on the first type of attention head vector for the target word, after determining the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy, it may further include: If the instruction channel number is equal to 1, then the vector length in the vector register is adjusted according to the half-rotation dimension; The attention head data is read from the first-level cache using a step-by-step reading method to obtain the first rotation component and the second rotation component of the current attention head vector. Based on the first rotation component, the second rotation component, and the rotation coefficient vector after broadcasting, the first output component or the second output component of the current attention head vector is calculated.

[0109] Optionally, if the number of instruction channels is equal to 1, the SIMD instruction will only process the rotation position encoding calculation of 1 attention head vector. In this case, the effective vector length in the vector register can be adjusted to half the rotation dimension. That is, only the first half of the rotation dimension elements of the vector register are used, and the remaining elements are masked to match the number of elements that need to be processed, thus avoiding invalid calculations.

[0110] Optionally, a step-load instruction with a step size of 2 can be executed from the even-numbered starting address of the current attention head vector to read all the first rotation components of the attention head vector, and then the same step-load instruction can be executed from the odd-numbered starting address to read all the second rotation components.

[0111] S270. When the data type of the first output component and the second output component is a 16-bit floating-point number, convert each element in the first output component and the second output component into a 16-bit unsigned integer, and after conversion, expand each element in the first output component and the second output component from a 16-bit unsigned integer to a 32-bit unsigned integer.

[0112] S280. Shift each expanded element in the second output component left by 16 bits and then perform a bitwise OR operation with each expanded element in the first output component to generate a concatenated vector.

[0113] S290. Write the concatenated vector into global memory to interleave the elements of the first output component and the second output component in global memory.

[0114] The technical solution of this invention, by calculating the maximum number of vectors that a single computation instruction can process and by using a multi-instruction channel loop expansion method to process attention head vectors, can process multiple attention head vectors simultaneously in a single SIMD instruction, significantly improving vector register utilization and reducing computational waste. By prefetching rotation coefficient vectors into the L1 cache, using multi-way loop expansion to mask memory access latency, and broadcasting rotation coefficient vectors to match the processing width of the SIMD instruction, zero-wait data provision is achieved, reducing memory access overhead for rotation coefficient vectors and avoiding redundant data movement. By designing a 16-bit floating-point bitwise operation merge write-back, two straddle storage operations can be merged into a single normal storage operation, halving the number of storage instructions and improving overall computational efficiency. By dynamically segmenting terms according to the L1 cache capacity and using a dual-channel data movement engine to asynchronously execute data movement, the pipeline of data movement and computation can overlap, maximizing resource utilization.

[0115] Example 3 Figure 3 This is a schematic diagram of a rotational position encoding acceleration device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data transfer module 310, a loop calculation strategy determination module 320, a calculation round iteration module 330, a multi-instruction channel calculation module 340, and a merge-write-back module 350.

[0116] The data transport module 310 is used to segment the lexical units allocated to the current thread, obtain at least one lexical unit block, and when the processing of the target lexical unit block is started, transport the attention head data and rotation coefficient vector of each lexical unit in the target lexical unit block to the first-level cache.

[0117] The loop calculation strategy determination module 320 is used to determine the multi-round loop calculation strategy based on the maximum number of vectors that a single calculation instruction can process and the attention head size of a single word, and broadcast the rotation coefficient vector of each word in the first-level cache.

[0118] The calculation round iteration module 330 is used to determine the number of instruction channels in the current calculation round according to the multi-round cyclic calculation strategy when performing rotation position encoding on the first type of attention head vector of the target word.

[0119] The multi-instruction channel calculation module 340 is used to read the attention head data in the first-level cache by step reading if the number of instruction channels is greater than 1, obtain the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel, and calculate the first output component or the second output component of each attention head vector in each instruction channel according to the first rotation component, the second rotation component and the rotation coefficient vector after broadcast.

[0120] The merge write-back module 350 is used to generate a concatenated vector based on the first output component and the second output component of the attention head vector, and write the concatenated vector into global memory so that the elements in the first output component and the second output component are cross-stored in global memory.

[0121] The technical solution of this invention involves segmenting the lexical units allocated to the current thread to obtain at least one lexical unit block. When processing a target lexical unit block is initiated, the attention head data and rotation coefficient vectors of each lexical unit in the target lexical unit block are moved to the first-level cache. A multi-round cyclic calculation strategy is determined based on the maximum number of vectors that a single calculation instruction can process and the size of the attention head of a single lexical unit, and the rotation coefficient vectors of each lexical unit are broadcast in the first-level cache. When encoding the rotation position of the first type of attention head vector for the target lexical unit, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, a step-by-step reading method is used to read the number of attention heads from the first-level cache. According to the method, the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained, and the first output component or the second output component of each attention head vector is calculated in each instruction channel. Based on the first output component and the second output component, a concatenated vector is generated and written into global memory. By interleaving the elements in the first output component and the second output component in global memory, the utilization of vector registers can be greatly improved, computational waste can be reduced, the memory access overhead of rotation coefficient vector can be reduced, duplicate data handling can be avoided, the number of write-back stage storage instructions can be effectively reduced, the overall computational efficiency can be improved, and the pipeline of data handling and computation can be overlapped to maximize the utilization of hardware resources.

[0122] Based on the above format example, a maximum vector number calculation module can also be included, used for: Based on the vector register width and the target data type used in the rotation position encoding, determine the maximum number of elements that a single computation instruction can process in parallel, and determine the half-rotation dimension based on the rotation dimension of the rotation position encoding. Based on the maximum number of elements and the semi-rotation dimension, determine the maximum number of vectors that a single computation instruction can process.

[0123] Based on the above format example, the data transport module 310 may include a word segmentation unit and a word data transport unit. Lexical segmentation units can be specifically used for: Based on the maximum number of vectors that a single computation instruction can process, the size of the attention head of a single word, and the capacity of the first-level cache, multiple storage areas are divided in the first-level cache; Based on the storage area partitioning results and the rotation dimension of the rotation position encoding, determine the number of target words and determine whether the number of target words is greater than or equal to the total number of words allocated to the current thread. If so, the tokens allocated to the current thread will be divided into a token block; If not, then based on the number of target lexical units, the lexical units allocated to the current thread are segmented to obtain multiple lexical unit blocks.

[0124] Based on the above format example, the attention header data includes multiple query header vectors and multiple key header vectors; The term data transport unit can be specifically used for: Through two data transfer engine channels, the query header vector and key header vector of each word in the target word block are transferred from global memory to the query header vector storage area and key header vector storage area in the first-level cache, respectively. Through multiple rounds of iterative loops, the rotation coefficient vectors of each word within the target word block are moved from global memory to a temporary storage area in the first-level cache in batches. In each round of iterative loop, the rotation coefficient vectors of the specified word are moved by multiple parallel loading instructions. The rotation coefficient vectors include a first rotation coefficient vector and a second rotation coefficient vector.

[0125] Based on the above format example, the loop calculation strategy determination module 320 can be specifically used for: Based on the maximum number of vectors that a single instruction can process and the preset maximum number of instruction channels, determine the first number of vectors that can be calculated using all instruction channels in a single round; Based on the attention head size of a single word and the number of the first vectors, determine at least one full instruction channel computation round and a first remaining vector that cannot be computed by the full instruction channel computation rounds; If the first remaining vector is greater than the maximum number of vectors that a single instruction can process, then based on the maximum number of vectors that a single instruction can process and the first remaining vector, the target number of computing channels in the multi-instruction channel computing round is determined, and the second remaining vector that cannot be calculated in the multi-instruction channel computing round is determined. Based on the second remaining vector, at least one single-instruction single-vector calculation round is determined.

[0126] Based on the above format example, a single-channel calculation module can also be included for: If the instruction channel number is equal to 1, then the length of the vector in the vector register is adjusted according to the half-rotation dimension; The attention head data is read from the first-level cache using a step-by-step reading method to obtain the first rotation component and the second rotation component of the current attention head vector. Based on the first rotation component, the second rotation component, and the rotation coefficient vector after broadcasting, the first output component or the second output component of the current attention head vector is calculated.

[0127] Based on the above format example, the merge write-back module 350 can be specifically used for: When the data type of the first output component and the second output component is 16-bit floating-point number, each element in the first output component and the second output component is converted into a 16-bit unsigned integer, and after the conversion, each element in the first output component and the second output component is extended from a 16-bit unsigned integer to a 32-bit unsigned integer. After shifting each expanded element in the second output component 16 bits to the left, perform a bitwise OR operation with each expanded element in the first output component to generate a concatenated vector.

[0128] The rotational position encoding acceleration device provided in the embodiments of the present invention can execute the rotational position encoding acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0129] Example 4 Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0130] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the rotation position encoding acceleration method described in the embodiments of the present invention. That is: The tokens allocated to the current thread are segmented to obtain at least one token block. When the processing of the target token block is started, the attention header data and rotation coefficient vector of each token in the target token block are moved to the first-level cache. Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, a multi-round loop computation strategy is determined, and the rotation coefficient vectors of each word are broadcast in the first-level cache. When performing rotational position encoding on the first type of attention head vector of the target word, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, the attention head data is read from the first-level cache using the step-read method, and the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel are obtained. In each instruction channel, the first output component or the second output component of each attention head vector is calculated based on the first rotation component, the second rotation component and the rotation coefficient vector after broadcast. Based on the first and second output components of the attention head vector, a concatenated vector is generated and written into global memory to interleave the elements of the first and second output components in global memory.

[0133] In some embodiments, the rotation position encoding acceleration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the rotation position encoding acceleration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the rotation position encoding acceleration method by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0139] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0140] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating rotational position encoding, characterized in that, include: The tokens allocated to the current thread are segmented to obtain at least one token block. When the processing of the target token block is started, the attention header data and rotation coefficient vector of each token in the target token block are moved to the first-level cache. Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head of a single word, a multi-round loop computation strategy is determined, and the rotation coefficient vectors of each word are broadcast in the first-level cache. When performing rotational position encoding on the first type of attention head vector of the target word, the number of instruction channels for the current calculation round is determined according to the multi-round cyclic calculation strategy. If the number of instruction channels is greater than 1, the attention head data is read from the first-level cache using the step-read method to obtain the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel. In each instruction channel, the first output component and the second output component of each attention head vector are calculated based on the first rotation component, the second rotation component and the rotation coefficient vector after broadcast. Based on the first and second output components of the attention head vector, a concatenated vector is generated and written into global memory to interleave the elements of the first and second output components in global memory.

2. The method according to claim 1, characterized in that, Before segmenting the tokens allocated to the current thread and obtaining at least one token block, the process also includes: Based on the vector register width and the target data type used in the rotation position encoding, determine the maximum number of elements that a single computation instruction can process in parallel, and determine the half-rotation dimension based on the rotation dimension of the rotation position encoding. Based on the maximum number of elements and the semi-rotation dimension, determine the maximum number of vectors that a single computation instruction can process.

3. The method according to claim 1, characterized in that, The tokens allocated to the current thread are segmented to obtain at least one token block, including: Based on the maximum number of vectors that a single computation instruction can process, the size of the attention head of a single word, and the capacity of the first-level cache, multiple storage areas are divided in the first-level cache; Based on the storage area partitioning results and the rotation dimension of the rotation position encoding, determine the number of target words and determine whether the number of target words is greater than or equal to the total number of words allocated to the current thread. If so, the tokens allocated to the current thread will be divided into a token block; If not, then based on the number of target lexical units, the lexical units allocated to the current thread are segmented to obtain multiple lexical unit blocks.

4. The method according to claim 1, characterized in that, The attention header data includes multiple query header vectors and multiple key header vectors; The attention header data and rotation coefficient vectors of each word in the target word block are moved to the first-level cache, including: Through two data transfer engine channels, the query header vector and key header vector of each word in the target word block are transferred from global memory to the query header vector storage area and key header vector storage area in the first-level cache, respectively. Through multiple rounds of iterative loops, the rotation coefficient vectors of each word within the target word block are moved from global memory to a temporary storage area in the first-level cache in batches. In each round of iterative loop, the rotation coefficient vectors of the specified word are moved by multiple parallel loading instructions. The rotation coefficient vectors include a first rotation coefficient vector and a second rotation coefficient vector.

5. The method according to claim 1, characterized in that, Based on the maximum number of vectors that a single computation instruction can process and the size of the attention head for a single word, a multi-round iterative computation strategy is determined, including: Based on the maximum number of vectors that a single instruction can process and the preset maximum number of instruction channels, determine the first number of vectors that can be calculated using all instruction channels in a single round; Based on the attention head size of a single word and the number of the first vectors, determine at least one full instruction channel computation round and a first remaining vector that cannot be computed by the full instruction channel computation rounds; If the first remaining vector is greater than the maximum number of vectors that a single instruction can process, then based on the maximum number of vectors that a single instruction can process and the first remaining vector, the target number of computing channels in the multi-instruction channel computing round is determined, and the second remaining vector that cannot be computed in the multi-instruction channel computing round is determined. Based on the second remaining vector, at least one single-instruction single-vector calculation round is determined.

6. The method according to claim 2, characterized in that, When performing rotational position encoding on the first type of attention head vector of the target word, after determining the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy, the method further includes: If the instruction channel number is equal to 1, then the vector length in the vector register is adjusted according to the half-rotation dimension; The attention head data is read from the first-level cache using a step-by-step reading method to obtain the first rotation component and the second rotation component of the current attention head vector. Based on the first rotation component, the second rotation component, and the rotation coefficient vector after broadcasting, the first output component or the second output component of the current attention head vector is calculated.

7. The method according to claim 1, characterized in that, Based on the first and second output components of the attention head vector, a concatenated vector is generated, including: When the data type of the first output component and the second output component is 16-bit floating-point number, each element in the first output component and the second output component is converted into a 16-bit unsigned integer, and after the conversion, each element in the first output component and the second output component is extended from a 16-bit unsigned integer to a 32-bit unsigned integer. After shifting each expanded element in the second output component 16 bits to the left, perform a bitwise OR operation with each expanded element in the first output component to generate a concatenated vector.

8. A rotational position encoding acceleration device, characterized in that, include: The data transport module is used to segment the lexical units allocated to the current thread, obtain at least one lexical unit block, and when the processing of the target lexical unit block is started, transport the attention head data and rotation coefficient vector of each lexical unit in the target lexical unit block to the first-level cache. The loop calculation strategy determination module is used to determine the multi-round loop calculation strategy based on the maximum number of vectors that a single calculation instruction can process and the size of the attention head of a single word, and broadcast the rotation coefficient vector of each word in the first-level cache. The calculation round iteration module is used to determine the number of instruction channels for the current calculation round according to the multi-round cyclic calculation strategy when performing rotation position encoding on the first type of attention head vector of the target word. The multi-instruction channel calculation module is used to read the attention head data in the first-level cache by step reading if the number of instruction channels is greater than 1, obtain the first rotation component and the second rotation component of multiple attention head vectors in each instruction channel, and calculate the first output component and the second output component of each attention head vector in each instruction channel according to the first rotation component, the second rotation component and the rotation coefficient vector after broadcast. The merge-write-back module is used to generate a concatenated vector based on the first and second output components of the attention head vector, and write the concatenated vector into global memory so that the elements in the first and second output components are cross-stored in global memory.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the rotation position encoding acceleration method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the rotation position encoding acceleration method according to any one of claims 1-7.

Images