Method for performing attention calculation using circuit arrangement and circuit arrangement

Abstract

A method for performing attention calculation using a circuit device, the circuit device, an electronic apparatus, and a non-transitory computer readable storage medium, the circuit device including a first cache module configured to cache first input data; the second cache module is configured to cache second input data; the transpose module is configured to receive the first input data from the first cache module and output transpose data; and the kernel module is configured to receive the transposed data from the transposed module, receive the second input data from the second cache module, execute kernel calculation by using the transposed data and the second input data, and input calculation result data to the first cache module. According to the embodiment of the invention, the transpose module is arranged behind the first cache module, so that a plurality of computing nodes in the attention computing process are merged in a data stream manner to form a data stream path for parallel execution, and the performance and efficiency of attention computing are improved.

Description

Technical Field

[0001] This invention relates to the field of integrated chip technology, and more specifically, to a method for performing attention computation using circuitry, as well as circuitry, electronic devices, and non-transitory computer-readable storage media. Background Technology

[0002] In recent years, large language models based on the transformer architecture have developed rapidly. The scale of large language models is getting larger and larger, and their performance is getting better and better. Dialogue applications based on various large language models are becoming more and more widespread.

[0003] Due to the immediacy of dialogue applications, the model has very high latency requirements for the inference process. Large language models are themselves very large in scale, and the inference process has a certain degree of sparsity. Therefore, the operation of large language models on hardware requires extremely high memory access bandwidth and computing power.

[0004] The core of the Transformer architecture is the attention module, whose structure is as follows: Figure 1 As shown, the corresponding calculation formula is shown in formula (1).

[0005] (1)

[0006] In this model, Q (Query), K (Key), and V (Value) are inputs calculated through three similar linear layers. The Q and K branches embed positional information using Rotation-Based Position Encoding (RoPE). Then, matrix multiplication (BMM) is performed on the Q and K branches, followed by a softmax operation to extract feature relevance. Finally, a matrix multiplication BMM is performed with the V branch to obtain the Attention output. In large language models for dialogue applications, the data from the K and V branches is typically stored as historical data. This data is then concatenated with new inputs and incorporated into the Attention calculation process, enabling the model to remember historical information. This characteristic of the algorithm model necessitates the allocation of significant external storage space for caching Kcache and Vcache data during inference, and the Kcache and Vcache data in the external storage space need to be read and written each time the algorithm model generates output.

[0007] Given this characteristic of the Attention structure, taking the V branch as an example, as the accelerator executes each operation node sequentially according to the algorithm graph of the Attention structure and outputs the data to external storage (DDR) for caching, it reads the data from external storage and back into the accelerator for computation when executing the next operation. This entire process requires frequent reads and writes to external storage. Therefore, as the dialogue length increases, the size of the Vcache cached data also increases rapidly, while the computational load of linear operations during inference and the subsequent BMM increases relatively little. This makes the data transfer between external storage and the accelerator of the Vcache data a bottleneck in the execution of the Attention structure. Summary of the Invention

[0008] The present invention aims to provide a method for performing attention computation using circuit devices, as well as circuit devices, electronic devices, and non-transient computer-readable storage media, to solve the problem of low computational efficiency caused by frequent reading and writing of external data during the attention computation process.

[0009] According to one aspect of the present invention, a circuit device for attention calculation is provided, comprising: The first cache module is configured to cache the first input data; The address generator is configured to control the order in which the first cache module reads and writes data based on the generated read / write addresses. The second cache module is configured to cache the second input data; The transpose module is configured to receive the first input data from the first cache module and output transposed data; The kernel module is configured to receive the transposed data from the transposed module, receive the second input data from the second cache module, perform kernel calculations using the transposed data and the second input data, and input the calculation result data into the first cache module.

[0010] According to some embodiments, the circuit device further includes a quantization module. The quantization module is configured to receive the first input data output by the first cache module or the calculation result data output by the kernel module to perform quantization operations, and output the quantization result to an external memory.

[0011] According to some embodiments, the circuit device further includes a first data selector. The kernel module is further configured to input the calculation result data into the first cache module through the first data selector; The quantization module is further configured to output the quantization result to an external memory via the first data selector.

[0012] According to some embodiments, the circuit device further includes a second data selector. The kernel module is further configured to receive the transposed data from the transposed module via the second data selector, or to receive the first input data from the first cache module.

[0013] According to some embodiments, the circuit device further includes a third data selector. The quantization module is further configured to receive the first input data output by the first cache module or the calculation result data output by the kernel module through the third data selector to perform quantization operation, and output the quantization result to external memory.

[0014] According to some embodiments, the address generator is further configured to generate read / write addresses in the order of transpose operations, and store the first input data into the first cache module according to the read / write addresses.

[0015] According to one aspect of the present invention, a method for performing attention calculation using a circuit device as described in any of the preceding embodiments is provided, comprising: The kernel module is used to perform linear operations on the first input data and the second input data to obtain a linear calculation result. Following the order of transpose operations, the calculation results are stored in the first cache module using the address generator.

[0016] According to some embodiments, the method further includes: The transpose module is used to perform a transpose operation on the calculation results in the first cache module to obtain transpose result data; The quantization module is used to perform quantization calculations on the transposed result data to obtain quantized result data; The quantization result data is output to an external storage space to perform a data splicing operation in the external storage space.

[0017] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform the transpose implementation method as described in any of the preceding embodiments.

[0018] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to perform the method as described in any of the preceding embodiments.

[0019] According to an embodiment of the present invention, by setting a transpose module after the first cache module, multiple computing nodes in the attention computing process are merged into a data flow path for parallel execution using a data flow approach, thereby improving the performance and efficiency of attention computing.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The above and other objectives, features, and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings.

[0022] Figure 1 A schematic diagram of the attention module calculation process is shown.

[0023] Figure 2a A schematic diagram of a prior art circuit device for attention calculation is shown.

[0024] Figure 2b A schematic diagram of a V-branch node structure for attention computation in the prior art is shown.

[0025] Figure 2c A schematic diagram of a Vcache data operation process in the prior art is shown.

[0026] Figure 3 A circuit device block diagram for attention calculation according to an exemplary embodiment of the present invention is shown.

[0027] Figure 4a A schematic diagram of a circuit structure for attention calculation according to an exemplary embodiment of the present invention is shown.

[0028] Figure 4b A schematic diagram of a simplified V-branch operation for attention calculation according to an exemplary embodiment of the present invention is shown.

[0029] Figure 4c A schematic diagram of a Vcache data operation process according to an example embodiment of the present invention is shown.

[0030] Figure 5 A flowchart illustrating a method for performing attention calculation using a circuit device according to an exemplary embodiment of the present invention is shown.

[0031] Figure 6 A flowchart illustrating another method for performing attention calculation using a circuit device according to an exemplary embodiment of the present invention is shown.

[0032] Figure 7 A schematic diagram of a simplified main branch operation process for attention calculation according to an exemplary embodiment of the present invention is shown.

[0033] Figure 8 A schematic diagram of a simplified K-branch operation for attention calculation according to an exemplary embodiment of the present invention is shown.

[0034] Figure 9 A data stream chip according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same drawings in the figures show the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0039] Figure 2a A schematic diagram of a prior art circuit device for attention calculation is shown, such as... Figure 2aAs shown, the on-chip first cache unit buf0 and second cache unit buf1 are used to cache the input data when the accelerator runs algorithm operations. For binocular operations, buf0 and buf1 each cache one input data path. For example, for linear operations, buf0 caches feature data and buf1 caches weight data. At the same time, buf0 and buf1 output the cached data to the kernel operation module for calculation according to the data arrangement format corresponding to the algorithm operation. The kernel operation module is the operation unit used to perform the calculation process of various algorithm operations. The dynamic quantization unit (DynQuant) is used to quantize floating-point data such as BF16 / FP16 / FP32 into INT8 / MXINT8 data. The direct memory access (DMA) unit is used to read data from the device memory (Double Data Rate, DDR) into the accelerator and write the results from the accelerator into the device memory.

[0040] When this hardware architecture is used to perform V-branch calculations, the corresponding V-branch node structure is as follows: Figure 2b As shown, this corresponds to the vcache data operation process. Figure 2c As shown, specifically, the linear node performs a linear operation on the input data corresponding to the current token to obtain the corresponding V data, which corresponds to... Figure 2c The leftmost row of vcache data is marked with a "New" tag. Here, "token" refers to a word or phrase in a sentence input by the user during the dialogue task, or a word or phrase in the response generated by the algorithm model. When using a multi-head attention structure, the transpose node transposes the result of the linear operation, moving the Head dimension to the outermost dimension of the data. Then, the Concat node concatenates the new V data generated by the linear operation with the historical vcache data in external storage to obtain the updated vcache data, such as... Figure 2c As shown on the far left, the rows corresponding to 0-5 in the token represent historical data. Concatenating these rows with the row corresponding to 6 in the token yields the latest vcache data. Then, the transpose node transposes the updated vcache data, resulting in the following... Figure 2c The transposed vcache data shown in the middle has changed the token sequence direction to the row direction.

[0041] In some embodiments, Figure 2aThe DynQuant module in the accelerator performs quantization on the transposed vcache data, converting it from high-precision BF16 or FP16 types to INT8 or MXINT8 types for matrix multiplication. When the accelerator performs inference tasks on the algorithm model, it typically uses quantized data types to accelerate computationally intensive tensor operations such as linear and matrix multiplication, thus improving performance. Therefore, the vcache data is first quantized before matrix multiplication. Since the vcache data includes continuously updated data, it needs to be quantized each time it is updated, and the quantization direction is along the token sequence. Figure 2c The directions of the tokens {0, 1, 2, ...} marked above. Finally, the matrix calculation BMM node performs matrix multiplication on the data after the Softmax operation in the Attention structure and the vcache data to obtain the calculated Attention result.

[0042] This existing solution has the following problems: 1) Each algorithm node in the Attention structure needs to be executed separately, resulting in low overall execution performance and utilization. 2) The execution process of the K branch and V branch requires frequent reading and writing of DDR, especially vcache data. Each operation requires repeated reading and writing of cached historical data multiple times, and data transfer time becomes a bottleneck, affecting the inference performance and efficiency of the accelerator.

[0043] To simplify the execution process of the Attention structure, according to an embodiment of the present invention, multiple algorithm operation nodes in the Attention structure are merged into a single data flow path, thereby executing in parallel as a data stream. This improves the performance of the accelerator in performing inference tasks, and thus enhances the overall utilization of the AI ​​chip. Simultaneously, the merged data flow path reduces the number of times intermediate results from algorithm nodes are read from and written to external storage, thereby reducing data transfer between external storage and the accelerator, preventing data transfer from becoming a performance bottleneck for the Attention structure execution, and lowering the accelerator's bandwidth requirements for external storage.

[0044] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] Figure 3 A circuit device block diagram for attention calculation according to an exemplary embodiment of the present invention is shown, such as Figure 3The circuit device shown includes a first cache module 301, a second cache module 303, a transpose module 305, a kernel module 307, and an address generator 309.

[0046] According to an embodiment of the present invention, a first cache module 301 is configured to cache first input data; a second cache module 303 is configured to cache second input data; a transpose module 305 is configured to receive the first input data from the first cache module and output transposed data; a kernel module 307 is configured to receive the transposed data from the transpose module, receive the second input data from the second cache module, perform kernel calculations using the transposed data and the second input data, and input the calculation result data into the first cache module. To control the data read / write order of the first cache module 301, an address generator 309 is configured to control the order of data read / write operations of the first cache module based on the generated read / write addresses.

[0047] In some embodiments, an address generator is used to generate read / write addresses in the order of transpose operations, and the first input data is stored in the first cache module according to the read / write addresses.

[0048] It should be noted that kernel computation is based on a preset algorithm to perform corresponding calculations, and this invention does not limit the specific algorithm content of kernel computation.

[0049] According to some embodiments of the present invention, Figure 3 The circuit device shown also includes a quantization module, which is configured to receive first input data output by the first cache module or calculation result data output by the kernel module to perform a quantization operation and output the quantization result to an external memory. For example, the quantization module can be used to quantize transposed data, converting the data from high-precision BF16 or FP16 types to INT8 or MXINT8 types.

[0050] In some embodiments, the circuit device further includes a first data selector, and the kernel module is further configured to input the calculation result data into the first cache module through the first data selector; the quantization module is further configured to output the quantization result to an external memory through the first data selector.

[0051] In other embodiments, the circuitry further includes a second data selector, and the kernel module is further configured to receive the transposed data from the transposed module or the first input data from the first cache module via the second data selector.

[0052] In other embodiments, the circuit device further includes a third data selector, and the quantization module is further configured to receive first input data output by the first cache module or calculation result data output by the kernel module through the third data selector to perform quantization operations and output the quantization result to an external memory.

[0053] In some embodiments, the circuit device further includes an address generator, which controls the order in which the first cache module reads and writes data. For example, the address generator controls the order in which the first cache module writes data so that the stored data in the first cache module is stored in an order equivalent to the order after a transpose operation.

[0054] according to Figure 3 The embodiment shown improves the performance and efficiency of attention computation by setting a transpose module after the first cache module and merging multiple computing nodes in the attention computation process into a data flow path for parallel execution.

[0055] Figure 4a A schematic diagram of a circuit structure for attention calculation according to an exemplary embodiment of the present invention is shown. Figure 4a The circuit results shown are in Figure 2a Based on the above, it also includes a transpose unit and an address generator (AG). The transpose unit is used to transpose the data output by the buffer unit buf0, and the address generator (AG) is used to generate the read and write addresses of the corresponding data in buf0 according to the algorithm operation, so as to control the read and write order of the input data in buf0.

[0056] Figure 4b A schematic diagram of a simplified V-branch operation for attention calculation according to an exemplary embodiment of the present invention is shown. Figure 4c A schematic diagram of a Vcache data operation process according to an exemplary embodiment of the present invention is shown. Figure 4b As shown, firstly, adjust the data arrangement format of vcache data stored in DDR, and arrange the data according to... Figure 2c The quantized structure is stored as shown. The `transpose` and `quantize` nodes are then moved onto the `concat` node, resulting in... Figure 4bThe computation graph shown in the middle; then, the linear node and the first transpose node are merged, and the second transpose node, the quantize node, and the concat node are merged to obtain the simplified rightmost hardware computation graph; finally, the result data of the fusion of the linear node and the transpose node is cached in the on-chip cache static random-access memory (SRAM).

[0057] It should be noted that in this invention, vcache refers to intermediate data continuously used during the attention calculation process, not a specific storage location. Typically, vcache data is stored in external DDR memory. Here, on-chip SRAM refers to the cache within the accelerator hardware, i.e. Figure 4a The buf0 cache in the chip has the advantages of smaller storage space and faster read and write speed compared to external DDR storage. Since it is cached in the chip cache, there is no need to read and write from external DDR storage to the accelerator.

[0058] The following is based on Figure 4b For example, combined with Figure 4a The calculation process of the simplified V branch is explained in detail.

[0059] First, perform linear and transpose fusion operations on the input data x, which corresponds to... Figure 4a The data flow path in the process includes the following calculation process: Input data is read from the device memory DDR via Direct Memory Access (DMA) and written to buf0 for buffering; Weight data is read from DDR via DMA and written to buf1 for caching; Data is read from buffer units buf0 and buf1 respectively and output to the kernel module for linear operation; The kernel module outputs the result of the linear operation and writes it to buf0 for caching. During the writing process of buf0, the AG module controls the generation order of the write address (waddr) to realize the transpose operation of the first transpose node.

[0060] In this embodiment, the AG module can control the generation order of write addresses, thereby controlling the sequential writing of data to different locations in the on-chip cache and adjusting the data arrangement order. For example, if the original data arrangement order is D0, D1, D2, D3, D4, D5, D6, D7, when the data is input sequentially, by controlling the write address generation order to 0, 2, 4, 6, 1, 3, 5, 7 through the AG module, the data storage order in the on-chip cache can be adjusted to D0, D4, D1, D5, D2, D6, D3, D7.

[0061] Then, the merging operation of the second transpose node, quantize node, and concat node is performed, and its data flow path includes the following process: The data from the cached fusion node results is read from buf0 and output to the transpose module for transpose operation, according to... Figure 4c The method shown transposes the token sequence from the column direction to the row direction; The transposed result is output to the DynQuant module for quantization, which quantizes the data from types such as BF16 / FP16 to types such as INT8 / MXINT8. The DynQuant module outputs the quantization results and writes them to DDR via DMA. When writing the results to DDR, the DMA needs to be controlled to generate the DDR write address, and the results are concatenated with the historical data in DDR to achieve the concatenation operation of the Concat node.

[0062] Finally, the concatenated vcache data is read from DDR and the subsequent matrix multiplication (BMM) operation is performed.

[0063] It should be noted that the fusion operations in this embodiment are all implemented in a data stream manner. The data flows through the corresponding modules and executes the corresponding node operations in a pipeline manner. The multiple node operations are executed in parallel, thereby overlapping the time of multiple node operations into the time of one operation, which greatly improves the performance of attention inference task execution.

[0064] The data processing procedure corresponding to this process is as follows: Figure 4c As shown, firstly, the data after the Linear operation is transposed to the arrangement format used in BMM calculation; then, the data is quantized into data of type INT8 / MXINT8, etc.; finally, it is concatenated with the quantized historical data stored in DDR to obtain the updated vcache data, which is then used for subsequent BMM calculations.

[0065] Figure 5A flowchart illustrating a method for performing attention calculation using a circuit device according to an exemplary embodiment of the present invention is shown. The method includes steps S501 and S503, wherein S501-S503 can be used to perform... Figure 4b The merging operation of the linear node and the first transpose node shown.

[0066] like Figure 5 In step S501, the kernel module is used to perform a linear operation on the first input data and the second input data to obtain a linear calculation result. In step S503, the calculation result is stored in the first cache module using the address generator in the order of transpose operations.

[0067] Figure 6 A flowchart illustrating another method for performing attention calculation using a circuit device according to an exemplary embodiment of the present invention is shown. The method includes steps S505, S507, and S509 in addition to steps S501 and S503, wherein S501-S503 can be used to perform… Figure 4b The merging operation between the linear node and the first transpose node shown can be performed using S505~S509. Figure 4b The second transpose node, quantize node, and concat node are shown in the fusion operation.

[0068] like Figure 6 As shown, in step S505, the transpose module is used to perform a transpose operation on the calculation result in the first cache module to obtain transpose result data; In step S507, the quantization module is used to perform quantization calculation on the transposed result data to obtain quantized result data; In step S509, the quantization result data is output to an external storage space to perform a data splicing operation in the external storage space.

[0069] according to Figure 5 and Figure 6 The embodiment shown uses a data flow approach to merge multiple algorithm operation nodes in the Attention structure, especially the V branch, into a data flow path for parallel execution, thereby improving the performance and efficiency of the Attention structure inference task.

[0070] It should be noted here that... Figure 4a The circuit structure shown is not only suitable for fusion calculations on the V branch of the Attention structure, but can also be used for fusion optimization on other branches if they contain similar combinations of operation nodes. The optimization graph on the main branch is shown below. Figure 7As shown, the optimization process on the K branch is as follows: Figure 8 As shown.

[0071] like Figure 7 As shown, the RoPE nodes and transpose nodes, as well as the BMM nodes and transpose nodes on the Q branch, can be merged separately. The execution process can employ... Figure 5 The method shown is implemented.

[0072] like Figure 8 As shown, the RoPE node and transpose node, as well as the Quantize node and Concat node on the K branch can be merged respectively, and the execution process can be implemented using the methods shown in steps S507 to S509.

[0073] Figure 9 An electronic device according to an exemplary embodiment of the present invention is shown below. Figure 9 To describe an electronic device 200 according to this embodiment of the present invention. Figure 9 The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0074] like Figure 9 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0075] The storage unit stores program code, which can be executed by the processing unit 210 to perform the methods described in this specification according to various exemplary embodiments of the present invention. For example, the processing unit 210 can perform the methods shown above.

[0076] Storage unit 220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 2201 and / or cache memory 2202, and may further include read-only memory (ROM) 2203.

[0077] Storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0078] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0079] Electronic device 200 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0080] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions of the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of the present invention.

[0081] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0082] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0083] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0085] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0086] According to an embodiment of the present invention, a computer program is provided, comprising a computer program or instructions, which, when executed by a processor, can perform the methods described above.

[0087] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A circuit device for attention calculation, characterized in that, include: The first cache module is configured to cache the first input data; An address generator is configured to control the order in which the first cache module reads and writes data based on the generated read / write addresses. The second cache module is configured to cache the second input data; The transpose module is configured to receive the first input data from the first cache module and output transposed data; The kernel module is configured to receive the transposed data from the transposed module, receive the second input data from the second cache module, perform kernel calculations using the transposed data and the second input data, and input the calculation result data into the first cache module.

2. The circuit device according to claim 1, characterized in that, It also includes a quantization module. The quantization module is configured to receive the first input data output by the first cache module or the calculation result data output by the kernel module to perform quantization operations, and output the quantization result to an external memory.

3. The circuit device according to claim 2, characterized in that, It also includes the first data selector. The kernel module is further configured to input the calculation result data into the first cache module through the first data selector; The quantization module is further configured to output the quantization result to an external memory via the first data selector.

4. The circuit device according to claim 3, characterized in that, It also includes a second data selector. The kernel module is further configured to receive the transposed data from the transposed module via the second data selector, or to receive the first input data from the first cache module.

5. The circuit device according to claim 4, characterized in that, It also includes a third data selector. The quantization module is further configured to receive the first input data output by the first cache module or the calculation result data output by the kernel module through the third data selector to perform quantization operation, and output the quantization result to external memory.

6. The circuit device according to claim 1, characterized in that, The address generator is further configured to generate read and write addresses in the order of transpose operations, and store the first input data into the first cache module according to the read and write addresses.

7. A method for performing attention calculation using a circuit arrangement as described in any one of claims 1-6, characterized in that, include: The kernel module is used to perform linear operations on the first input data and the second input data to obtain a linear calculation result. Following the order of transpose operations, the calculation results are stored in the first cache module using the address generator.

8. The method according to claim 7, characterized in that, Also includes: The transpose module is used to perform a transpose operation on the calculation results in the first cache module to obtain transpose result data; The quantization module is used to perform quantization calculations on the transposed result data to obtain quantized result data; The quantization result data is output to an external storage space to perform a data splicing operation in the external storage space.

9. An electronic device, comprising: processor; as well as A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in claim 7 or 8.

10. A non-transitory computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in claim 7 or 8.

Images