Computing device, chip, board card and method

Abstract

The invention discloses a computing device, a chip, a board card and a method, and relates to the technical field of artificial intelligence. The computing device may be included in a combined processing device, which may also include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the other processing devices and used for storing data of the computing device and the other processing devices. According to the embodiment of the invention, the operation in the attention mechanism operation process is fused to obtain the fusion operation, and the fusion operation is executed through the convolution operation circuit, so that the operation efficiency of the computing device and the attention mechanism operation speed are improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to a computing device, a chip, a board, and a method for using the computing device to perform attention mechanism operations. Background Art

[0002] Currently, deep learning algorithms based on the attention mechanism and its variants have shown strong performance in tasks such as natural language, image processing, and assisted programming, and their applications have been widely popularized at an unprecedented rate. However, training and using this type of algorithm for reasoning involves a huge amount of computation, and how to improve the computational speed of the attention mechanism and its variants has become an important issue that limits the further development of this type of algorithm.

[0003] In order to meet the needs of high-performance computing, existing artificial intelligence (AI) processors have made some designs in the processor architecture, hardware, instructions, etc., so that AI processors can provide higher performance and lower power consumption when performing AI tasks.

[0004] In view of this, there is an urgent need to provide a solution that can improve the computing efficiency of the attention mechanism in the computing scenario based on the structural characteristics of the AI ​​processor, thereby improving the computing speed of the attention mechanism. Summary of the invention

[0005] In order to at least solve one or more technical problems described in the above background technology section, the present disclosure proposes the following technical solutions and multiple embodiments thereof.

[0006] In a first aspect, the present disclosure discloses a computing device comprising a control circuit and an operation circuit, wherein: the control circuit is configured to control the operation circuit to perform an attention mechanism operation in a block loop; and the operation circuit comprises: a convolution operation circuit, configured to perform an original score calculation in the attention mechanism operation and an anti-overflow sub-operation in a normalization calculation to obtain a first intermediate result; a normalization calculation circuit, configured to perform the normalization processing in the normalization calculation on the first intermediate result to obtain a second intermediate result; and a matrix multiplication circuit, configured to multiply the second intermediate result with a corresponding value vector V to obtain an output.

[0007] In a second aspect, the present disclosure discloses a chip, comprising the computing device according to the first aspect.

[0008] In a third aspect, the present disclosure discloses a board comprising the chip according to the second aspect.

[0009] In a fourth aspect, the present disclosure discloses a method for performing attention mechanism operations using the computing device described in the first aspect.

[0010] By utilizing the computing device, chip, board and method disclosed in the present disclosure, the raw score calculation in the attention mechanism operation process and the anti-overflow sub-operation in the normalization calculation can be fused to obtain a fused operation, and a convolution operation circuit is used to perform the fused operation, thereby reducing the number of instructions involved in the attention mechanism operation process and improving the operating efficiency of the computing device and the attention mechanism operation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0012] Figure 1 An exemplary structural diagram of a board in some embodiments of the present disclosure is shown.

[0013] Figure 2 An exemplary structural diagram of a combined processing device in some embodiments of the present disclosure is shown.

[0014] Figure 3 An exemplary internal structure diagram of a computing device in some embodiments of the present disclosure is shown.

[0015] Figure 4 An exemplary structural diagram of a processor core in some embodiments of the present disclosure is shown.

[0016] Figure 5 An exemplary schematic diagram showing a processor core writing data to a processor core of another cluster in some embodiments of the present disclosure is shown.

[0017] Figure 6 An exemplary schematic diagram of a two-layer three-stage pipeline in some embodiments of the present disclosure is shown.

[0018] FIG7( a ) shows an exemplary schematic diagram of the attention mechanism calculation process in some embodiments of the present disclosure.

[0019] FIG7( b ) shows an exemplary schematic diagram of anti-overflow softmax in some embodiments of the present disclosure.

[0020] FIG8( a ) shows an exemplary schematic diagram of the blocking operation in the Flash-Attention algorithm.

[0021] FIG8( b ) shows an exemplary schematic diagram of the execution process of an inner loop in the Flash-Attention algorithm.

[0022] Fig. 9An exemplary structural diagram of a computing device in some embodiments of the present disclosure is shown.

[0023] Fig.10 An exemplary structural diagram of a convolution operation circuit in some embodiments of the present disclosure is shown.

[0024] Fig.11 An exemplary schematic diagram of the logical flow of attention mechanism operation in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0026] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0030] Exemplary Hardware Environment

[0031] Figure 1FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-on-chip, or a system on chip (SoC), which integrates one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud-based intelligent applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0032] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0033] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0034] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device (DRAM) 204 .

[0035] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0036] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0037] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0038] The storage device 204 may be DRAM, which is an off-chip memory used to store data to be processed. It is DDR and is usually 16G or larger in size. It is used to save data of the computing device 201 and / or the processing device 203.

[0039] Figure 3 The internal structure diagram of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the computing device 201 is composed of a hierarchy of system on a chip-cluster-processor core.

[0040] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 , and multiple clusters 305 .

[0041] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading data from outside the chip or writing data to outside the chip. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as examples in the figure. With the development of hardware, the computing device 201 disclosed in the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0042] At the cluster level, Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0043] The figure shows four processor cores 306 as an example, but the present disclosure does not limit the number of processor cores 306. Figure 4 Each processor core 306 includes three modules: a control module 41 , a computing module 42 and a storage module 43 .

[0044] The control module 41 is used to coordinate and control the operation of the computing module 42 and the storage module 43 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the computing module 42 and the storage module 43 as control information.

[0045] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, such as matrix multiplication and convolution.

[0046] The storage module 43 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0047] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0048] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (cluster direct memory access, CDMA) 310 and a global direct memory access module (global direct memory access, GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0049] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0050] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0051] The CDMA 310 is used to control the memory access of the SRAM 308 between different clusters 305 in the same computing device 201 . Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of CDMA310. In this application scenario, the same computing device includes multiple clusters. For the convenience of explanation, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 each include multiple processor cores. For the convenience of explanation, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0052] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. Then the slave sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0053] Back to Figure 3, GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODMA 433; the second channel is to first transfer data between DRAM 204 and SRAM 308 through GDMA 311, and then transfer data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present disclosure can select a data transmission channel according to the hardware conditions thereof.

[0054] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, this disclosure regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure.

[0055] An important reason why the computing device 201 has strong computing power is the three-level computing hierarchy of system on chip - cluster - processor core, combined with a three-layer memory design such as DRAM-SRAM-NRAM / WRAM, which enables data to be cached and calculated at the appropriate level to form a sufficient pipeline.

[0056] When the computing device 201 performs computing, the process can be divided into the following three phases: loading phase (load): loading data; computing phase (compute): moving data, computing, and moving intermediate results; and storing phase (store): storing results.

[0057] In more detail, in some embodiments, a two-layer three-stage pipeline may be used, such as Figure 6As shown, the first-level loading stage 601, calculation stage 602, and storage stage 603 occur in the cluster level. The first-level loading stage 601 is that the GDMA 330 loads data from the DRAM 204 to the SRAM 308, the first-level calculation stage 602 is that the cluster 305 calculates the loaded on-chip unit map and generates calculation results, and the first-level storage stage 603 is that the GDMA 330 stores the calculation results from the SRAM 308 back to the DRAM 204.

[0058] Since the cluster 305 includes multiple processor cores 306, the first layer calculation stage 602 actually divides the on-chip unit map into corresponding sub-maps through the storage core 307, and broadcasts them to at least one processor core 306 for calculation. Therefore, the second layer three-level waterline occurs in the processor core 306. In more detail, the second layer loading stage 604 is to load the sub-map from the SRAM 308 to the NRAM 431 by the MVDMA 434, the second layer calculation stage 605 is to move the sub-map and the sub-weight to the operation module 42 for calculation, and then move the intermediate result back to the NRAM 431, and the second layer storage stage 606 is to store the intermediate result from the NRAM 431 back to the SRAM 308 by the MVDMA 434.

[0059] The first-level pipeline means that the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage stage 603 can be performed in parallel. Now, taking the case where the same cluster 305 wants to process the j-th on-chip unit diagram, the j+1-th on-chip unit diagram and the j+2-th on-chip unit diagram as an example, first, the j-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 601. Then, the j-th on-chip unit diagram is calculated in the first-level calculation stage 602, and the first calculation result is transferred back to the SRAM 308. While the j-th on-chip unit diagram is being calculated, the j+1-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 607. When the first calculation result is stored back to DRAM 204 in the first level storage-back stage 603, the j+1th on-chip unit map is calculated in the first level calculation stage 608, and the second calculation result is transferred back to SRAM 308, and the j+2th on-chip unit map is loaded into SRAM 308 in the first level loading stage 610. The first level pipeline is rolled in this way.

[0060] In order to cooperate with the operation of the aforementioned pipeline, the SRAM 308 of this embodiment includes two storage spaces: a ping storage unit and a pong storage unit. The data flow is divided into three types according to the ping-pong attribute of the SRAM 308: input / output ping-pong (IOparity), input ping-pong (input parity), and no ping-pong (no parity). Input / output ping-pong can support parallel loading, calculation, and storage. To realize input / output ping-pong, the ping storage unit and the pong storage unit need to be completely equal, and are used for loading and storage respectively. Input ping-pong only supports storage and calculation in parallel, which will increase the time of transportation on the SRAM 308. Compared with input / output ping-pong, the ping storage unit and the pong storage unit do not need to be completely equal, but an additional cache (cache) of the same size as the storage space needs to be allocated. No ping-pong means that loading / storage and calculation are serial, and space does not need to be additionally allocated.

[0061] In order to implement the aforementioned first-level pipeline, the SRAM 308 of some embodiments has ping memory cells and pong memory cells of the same size to achieve the input / output ping-pong effect. Figure 6 For illustration, the storage area involved in the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage-back stage 603 of the j-th on-chip unit diagram is limited to the ping storage unit, while the storage area involved in the first-level loading stage 607, the first-level calculation stage 608 and the first-level storage-back stage 609 of the j+1-th on-chip unit diagram is limited to the pong storage unit, and the storage area involved in the first-level loading stage 610, the first-level calculation stage 611 and the first-level storage-back stage 612 of the j+2-th on-chip unit diagram is again limited to the ping storage unit, and in this way, the ping storage unit and the pong storage unit are used alternately for storage.

[0062] The second-level pipeline means that the second-level loading stage 604, the second-level calculation stage 605, and the second-level storage stage 606 can be processed in parallel. Take the case where the same processor core 306 wants to process the i-th sub-graph, the i+1-th sub-graph, and the i+2-th sub-graph in the j-th on-chip unit graph as an example. First, the i-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 604. Then, the i-th sub-graph is calculated in the second-level calculation stage 605 to generate the i-th intermediate result, and the i-th intermediate result is transferred back to the NRAM 431. At the same time, the i+1-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 613. The i-th intermediate result is stored back to the SRAM 308 in the second-level storage-back stage 606, and at the same time, the i+1-th sub-graph is calculated in the second-level calculation stage 614 to generate the i+1-th intermediate result, and the i+1-th intermediate result is moved back to the NRAM 431, and the i+2-th sub-graph is loaded into the NRAM 431 in the second-level loading stage 615.

[0063] Considering that the tasks of each cluster 305 are different and the completion times are naturally inconsistent, the synchronization module 304 of some embodiments may use synchronization barrier instructions to synchronize the task completion times to avoid timing errors.

[0064] Attention Mechanism Operation Scheme

[0065] In the attention mechanism calculation, three tensors or intermediate variables are involved: the query tensor Q (Query), the key tensor K (Key), and the value tensor V (Value). Q, K, and V are obtained by linearly transforming the input of the attention mechanism, that is, each input feature is projected onto the three tensors Q, K, and V. Simply put, the Q tensor is used to match other units, the K tensor is used to be matched by other units, and the V tensor represents the extracted information.

[0066] In the operation of the attention mechanism, the correlation between the Q tensor and the K tensor is first calculated, that is, a matrix multiplication operation is performed on the Q tensor and the K tensor to obtain the original score, which is also the similarity between the two tensors. Next, the original score is adjusted by scaling. Then, the scaled original score is normalized by softmax to obtain the attention score. Finally, the normalized attention score is used as a weight to weight the V tensor, that is, a matrix multiplication operation is performed with the V tensor to obtain the attention output.

[0067] The operation process of the above attention mechanism can be expressed as the following formula:

[0068]

[0069] Among them, Q, K, and V are the query Q tensor, key K tensor, and value V tensor mentioned above, respectively. The superscript T indicates transpose, and dk indicates the dimension of K.

[0070] In some embodiments, an anti-overflow softmax operation may be employed, which may be expressed as follows:

[0071]

[0072] Among them, z j is the j-th input to the softmax layer.

[0073] FIG7(a) shows an exemplary schematic diagram of the attention mechanism calculation process in some embodiments of the present disclosure. As shown in FIG7(a), the input of the attention mechanism is sequence data S, and the output is O. The shape of S can be (seq_len, d Embedding ) or (seq_len, d model ), and the shape of O is (seq_len, dmodel ). Among them, seq_len is the sequence length, that is, the number of elements in the input sequence; d Embedding is the length of the word embedding vector representing each element in the input sequence in the embedding layer of the model; d model is the length of the vector representing each element in the input sequence in the middle layer of the model. It can be understood that in FIG7(a), the shape of S is (4, 5) and the shape of O is (4, 4) for illustration. For those skilled in the art, different shapes can be selected according to actual conditions, and this disclosure does not limit this.

[0074] Continuing with Figure 7(a), the calculation of the attention mechanism mainly includes the following operations:

[0075] Linear transformation operation. Use three matrices W_Q, W_K, and WV to perform linear transformation on S, and obtain the query (Query, Q) matrix, key (Key, K) matrix, and value (Vavue, V) matrix; among which, the shapes of Q and K matrices are (seq_len, d k ), the shape of the V matrix is ​​(seq_len, d v ); In particular, in some embodiments, d k =d v .

[0076] Attention score calculation operation. Perform inner product operation on Q and K to obtain the original score (Logits) matrix; then perform softmax operation on Logits row by row to obtain the attention score (Score) matrix. As shown in the steps indicated by the dashed arrows in Figure 7(a), in some embodiments, after obtaining Logits, Logits are scaled. Specifically, Logits are divided by The softmax operation involves exponential operations on the elements in Logits, where the exponential operation e x When the value of x is relatively large, numerical overflow problems may occur. In some embodiments, an anti-overflow softmax operation may be used to calculate the attention score.

[0077] FIG7(b) shows an exemplary schematic diagram of anti-overflow softmax in some embodiments of the present disclosure. As shown in FIG7(b), firstly, the maximum value of the Logits matrix is ​​taken row by row to obtain the original score row maximum value row_max; the value of the corresponding row in row_max is subtracted from each element in Logits to obtain the anti-overflow Logits; then, an exponential operation is performed on each element in the anti-overflow Logits to obtain the exponential numerical matrix Exp; then, the sum of each row of Exp is calculated to obtain the vector row_sum; finally, each element in Exp is divided by the value of the corresponding row in row_sum to obtain the attention score Score.

[0078] Going back to Figure 7(a), we finally perform a matrix multiplication operation, that is, multiply Score by V to get the output O of the attention mechanism.

[0079] As mentioned in the background technology, deep learning algorithms based on attention mechanisms and their variants are becoming widely popular at an unprecedented rate. However, training and using this type of algorithm for reasoning involves a huge amount of computation. How to improve the computational speed of the attention mechanism and its variants has become an important issue that limits the further development of this type of algorithm. The computational complexity and memory consumption of the attention mechanism algorithm are proportional to the square of the input sequence length seq_len. When the sequence length continues to increase, the attention mechanism operator can easily become the performance bottleneck of the entire network. In response to this, the Flash-Attention algorithm uses the idea of ​​block calculation to execute the attention mechanism operation in a block loop, thereby effectively improving the running speed and performance of the attention mechanism.

[0080] Figure 8(a) shows an exemplary schematic diagram of the block operation in the Flash-Attention algorithm. As shown in Figure 8(a), for a shape of N×d model The matrix Q can be obtained by block division into q a×d model The block is denoted as Q 1 ~Q q ; For the shape of N×d model The matrices K and V can be obtained by dividing into p blocks with a shape of b×d model The blocks are denoted as K 1 ~K p 、V 1 ~V p After the block, based on the block Q i , K 1 ~K p 、V 1 ~V p , we can calculate the output O of the attention mechanism and Q i The corresponding block Oi The Flash-Attention algorithm can be implemented through two loops, where the loop variable of the outer loop is i, which is used to traverse Q i ; The loop variable of the inner loop is j, which is used to traverse K j 、V j ; The inner loop is based on block Q i , K 1 ~K p 、V 1 ~V p Calculate the output block O i .

[0081] Specifically, the operations performed by the inner loop include:

[0082] Initialize the elements of the vector row_max representing the maximum value of the original score row to all negative infinitesimals, the elements of the vector row_sum representing the sum of the rows to all 0, and i All elements in are 0.

[0083] Next, enter the loop and for the values ​​of j from 1 to p, follow the Q i , K j 、V j Calculate and update row_max, row_sum, and O i .

[0084] Final calculation O i =O i / row_sum.

[0085] FIG8( b ) shows an exemplary schematic diagram of the execution process of an inner loop in the Flash-Attention algorithm, where an inner loop is the above-mentioned Q i , K j 、V j Calculate and update row_max, row_sum, and O i As shown in Figure 8(b), the calculation process includes:

[0086] Inner product operation. That is, Q i , K j Do the inner product and get the Logits i，j .

[0087] Anti-overflow indexation operation. The first is to subtract the maximum value of the row, that is, Logits i，j Take the maximum value by row and get row_max i，j Then, take row_max row by row i，j, row_max, to obtain the current maximum value cur_max, that is: cur_max = max(row_max i，j , row_max), for Logits i，j Each element in is subtracted from the value of the corresponding row in cur_max; then the indexation operation is performed, that is, Logits i，j Each element in is exponentially increased to obtain exp i，j .

[0088] The sum operation. That is, exp i，j Find the sum of each row to get row_sum i，j .

[0089] Scaling factor calculation operation. That is, calculate the scaling factor scale = e row_max_cur_max .

[0090] Output block calculation operation. That is, calculation O i，j =exp i,j *V j .

[0091] Update operation. Including update O i O i *scale+O i，j , update row_max to cur_max and update row sum =row sum *scale+row_sum i，j .

[0092] With respect to the inner layer circulation, the inventors of the present disclosure have found that there are at least the following problems:

[0093] (1) The inner product operation in the inner loop is a matrix multiplication operation. The anti-overflow exponentiation operation after the inner product operation requires the matrix multiplication result to be subtracted from the row maximum value. This subtraction of the row maximum value operation will introduce additional processing time. This is because the existing operation modules are usually designed to include vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation. The matrix operation unit is responsible for the core calculation of deep learning algorithms, such as matrix multiplication. Different instructions are usually required in the computing device to indicate the operations performed by different types of operation units, that is, at least two instructions need to be executed to complete the process from inner product operation to subtraction operation. This involves the reading of multiple instructions, data access, and execution overhead, which reduces the operating efficiency of the computing device.

[0094] (2) The softmax operation involves the exp (exponential) calculation process. When the input data of exp is very small, the result after the exp operation is basically zero, and all subsequent calculation results are also zero. It can be considered that invalid calculations have been performed, resulting in a waste of computing power.

[0095] In view of this, in order to solve one or more technical problems such as low operating efficiency and waste of computing power of the computing device in the above-mentioned attention mechanism calculation scenario, the present disclosed embodiment proposes a computing device, which utilizes the convolution operation circuit provided in the computing device to integrate the calculation process that can only be implemented by multiple instructions in the Flash-Attention algorithm into one instruction, thereby improving the operating efficiency of the computing device and completing the attention mechanism calculation task at a faster operating speed.

[0096] It should be noted that in the disclosed embodiments, the attention mechanism computing task can exist in applications in various fields, and the data involved can be of various types. For example, some major application fields of the attention mechanism include: machine translation, sentiment analysis, text summarization, text classification, named entity recognition, etc. in natural language processing (NLP), and the corresponding data types can include text data, vocabulary embedding, sentence structure, etc.; facial recognition, image classification, target detection, image segmentation, etc. in computer vision (CV), and the corresponding data types can include pixel data, image features, etc.; intelligent assistants, automatic subtitle generation, speech to text conversion, etc. in speech recognition (ASR), and the corresponding data types can include audio signals, spectrograms, etc.; generative AI applications, etc. Generative AI refers to artificial intelligence technology that uses complex algorithms, models and rules to learn from large-scale data sets to create new original content, such as but not limited to the ability to create multiple types of content such as text, pictures, sounds, videos and codes. Accordingly, the input data processed by the attention mechanism computing task can be image data, audio data, video data, voice data, text data, document data, etc. The output of the attention mechanism computation task can include a likelihood score that an image belongs to a particular object category, a likelihood score that a document is about a particular topic, a likelihood score that a text segment in a target language is a correct translation of a text segment in a source language, or a likelihood score that a text segment is a correct transcription of a spoken utterance, etc.

[0097] Fig. 9 An exemplary schematic diagram of a computing device in some embodiments of the present disclosure is shown. Fig. 9As shown, the computing device 910 includes a control circuit 910 and an operation circuit 920, wherein the control circuit 910 is configured to control the operation circuit 920 to perform the attention mechanism operation in a block cycle. The operation circuit 920 includes: a convolution operation circuit 921, configured to perform the raw score calculation in the attention mechanism operation and the anti-overflow sub-operation in the standardization calculation to obtain a first intermediate result; a standardization calculation circuit 922, configured to perform the standardization processing in the standardization calculation on the first intermediate result to obtain a second intermediate result; and a matrix multiplication circuit 923, configured to multiply the second intermediate result with the corresponding value vector V to obtain an output.

[0098] As for the control circuit 910, it can be understood that its control operation circuit 920 performs the attention mechanism operation in a block cycle, which may specifically include: first, the input sequence S of the attention mechanism operation task is linearly transformed to obtain Q, K, and V, and the block operation is performed as shown in FIG. 8(a) above to obtain Q 1 ~Q q , K 1 ~K p 、V 1 ~V p ; Then set up two layers of loops to traverse all blocks of Q, K, and V, so as to calculate the output result O of the attention mechanism operation task. As for the setting of the two layers of loops, in some embodiments, it can be executed according to the first loop setting method described above, that is, the outer loop is used to traverse Q 1 ~Q q , the inner loop is used to traverse K 1 ~K p 、V 1 ~V p , where an outer loop is used to i , K 1 ~K p 、V 1 ~V p Calculate block O i In some other embodiments, the second loop setting method can be used, and the loop variable of the outer loop is i, which is used to traverse K 1 ~K p 、V 1 ~V p , and the loop variable of the inner loop is j, which is used to traverse Q 1 ~Q q For those skilled in the art, it is known that the second cycle setting method provided here is similar to the first cycle setting method, and will not be described in detail here.

[0099] The convolution operation circuit 921 can perform a convolution operation according to a convolution instruction. A complete convolution operation can be expressed as:

[0100]

[0101] That is, the output neuron (Output) is equal to the convolution of the input neuron (Input) and the weight (Weight) plus the bias (Bias). Specifically, the convolution operation is the process of using a convolution kernel to slide on the input matrix to calculate the output matrix. Input represents the input matrix, Weight represents the convolution kernel weight matrix, and Output represents the output matrix. Every time the convolution kernel slides to a local area of ​​the Input, sub The convolution operation is to combine Weight and I sub Perform element-wise multiplication and add the products to get an output value O sub This process can be seen as the sum of the multiplication of the corresponding elements of the two matrices, which is similar to the operation performed on each row of A and each column of B when performing the inner product operation on the two matrices A and B.

[0102] Next, taking the first loop setting as an example, in the process of performing the attention mechanism operation in a block loop, it is necessary to perform the block Q of Q i and K's block K j Perform inner product operation to get the original score Logits i,j , the subsequent anti-overflow indexation operation requires Logits i,j Execute the maximum value operation of the subtraction row, here record Logits i,j The row maximum value vector is row_max i,j , then, for Q i and K j Performing inner product operations and then reducing the row maximum value can be fused into:

[0103]

[0104] That is, after determining or estimating the Logits i,j The row maximum value vector is row_max i,j In the case of, the convolution operation circuit can be used to perform the original score calculation (ie, the inner product operation) and the anti-overflow sub-operation in the normalization calculation (ie, the subtraction of the original score row maximum value operation in the anti-overflow exponentiation operation) to obtain the first intermediate result (result 1 ).

[0105] Considering that the maximum value of the raw score row in the current round cannot be obtained when performing the raw score calculation (i.e., the inner product operation), the maximum value of the raw score row in the previous round can be used to participate in the calculation of the current round. In some embodiments, the attention mechanism operation task is set to the maximum value of the raw score row_max i,j Reference value row_max ref , that is to say, the first intermediate result is calculated by the following formula (3):

[0106]

[0107] It is understood that after obtaining the first intermediate result, the first intermediate result can be normalized using the normalization calculation circuit 922 to obtain the second intermediate result. Taking the above-mentioned first loop setting as an example, the process may include exponential operation on the first intermediate result to obtain the second intermediate result exp i，j matrix.

[0108] After obtaining the second intermediate result, the matrix multiplication circuit 923 can be used to multiply the second intermediate result with the block matrix of the corresponding value vector V to obtain an output. Continuing to use the above-mentioned first loop setting method as an example, after obtaining exp i，j After the matrix, the output block calculation operation can be performed: i，j =exp i,j *V j .

[0109] It can be understood that according to the technical solution of the above-mentioned embodiments of the present disclosure, the inner product operation of the Q and K blocks and the row subtraction maximum value in the process of executing the attention mechanism operation in a block loop are integrated to obtain a fused operation, and the fused operation is performed through a convolution operation circuit, thereby reducing the number of instructions involved in the attention mechanism operation process and improving the operating efficiency of the computing device and the attention mechanism operation speed.

[0110] Fig.10 An exemplary structural diagram of a convolution operation circuit in some embodiments of the present disclosure is shown. Fig.10 As shown, in some embodiments, the convolution operation circuit 921 may further include: a convolution circuit 924, configured to perform a matrix operation on the query vector Q and the key vector K to obtain an original score; and an addition circuit 925, configured to subtract a reference value of the maximum value of the original score row of the previous cycle from the original score to obtain a first intermediate result.

[0111] Based on what has been said before, the convolution circuit here can be used to perform The addition circuit can be used to perform the result 1 =result+(-row_max ref) operation. It is understandable that row_max ref As a pair of Logits i，j The row maximum value vector row_max i，j The estimated value of row_max can be adjusted adaptively as the program executes, for example, continuing to use the row_max of the previous loop in the current loop ref Or adjust row_max ref .

[0112] It is understandable that row_max is used in the previous embodiment. ref Estimating row_max i，j , in row_max ref When the value is set to be relatively small, numerical overflow may occur in the exponential operation. In this regard, in some embodiments, the normalization calculation circuit is further configured to: in response to the maximum value of the first intermediate result not exceeding the first set threshold, directly perform normalization processing on the first intermediate result; and / or in response to the maximum value of the first intermediate result being greater than the first set threshold, use the maximum value of the first intermediate result to correct the first intermediate result, and perform normalization processing on the corrected first intermediate result.

[0113] It is understandable that after getting the first intermediate result 1 Later, you can 1 Find the maximum value by row to get the vector max 1 =max(result 1 ), if max 1 If every element in is less than or equal to the first set threshold, there is no need to worry about value overflow and you can directly use result 1 Perform normalization; if the vector max i If there are elements in the result that are greater than the first set threshold, then result 1 Correction is performed to avoid numerical overflow. Regarding the setting of the first set threshold, the result 1 The result of exponential operation is exp i，j The data type used is set, that is, according to exp i，j The first set threshold is set according to the maximum value that the adopted data type can represent, and the present disclosure does not impose any limitation on this.

[0114] In some embodiments, the operation circuit is further configured to: in response to the maximum value of the first intermediate result being greater than a first set threshold, update the reference value of the original fractional row maximum value using the maximum value of the first intermediate result for use by the convolution operation circuit in the next cycle.

[0115] It is understandable that in order to prevent numerical overflow, the max 1 If there are elements greater than the set threshold in the attention mechanism operation task, the reference value row_max of the maximum value of the original score row set can be used ref Set to max 1 For example, if in max 1 There is max 1 [i]≥Set threshold, you can set row_max accordingly ref =max(max 1 ). Alternatively, you can also use the reference value row_max of the current raw score row maximum value ref Add max 1 For example, if in max 1 There is max 1 [i]≥Set threshold, you can set row_max accordingly ref =row_max ref +max(max 1 ). In this way, the reference value row_max of the maximum value of the original score row can be adaptively increased in each cycle during the block-by-block attention mechanism operation. ref In some embodiments, the first set threshold value is any one of the following: 0, 2, 4.

[0116] In some embodiments, the operation circuit may be further configured to: in response to the maximum value of the first intermediate result not exceeding the second set threshold, set the output of the block in the current cycle to 0.

[0117] It is understandable that in the result 1 Perform exponentialization to get exp i，j After the matrix, you need to convert exp i，j And the corresponding V matrix block V j Multiply to calculate the output block O i，j If the result 1 The values ​​in are very small, and the exp obtained by exponential operation is i，j The elements in the matrix will be approximately equal to 0, and these elements that are approximately equal to 0 will be equal to 0 in exp i，j With V j In the multiplication operation, it becomes an invalid value, and the multiplication operation can be considered as an invalid calculation, which wastes computing power. 1 Later, you can 1 Find the maximum value by row to get the vector max 1 =max(result1 ), and in max 1 When the maximum value of does not exceed the second set threshold, the output of the block in the current cycle is set to 0. In some embodiments, the second set threshold is determined based on the minimum value of the data type used in the operation. For example, when the data type is BFLOAT16, the second set threshold can be: -128.

[0118] It can be understood that the disclosed embodiment identifies invalid calculations in the attention mechanism operation task by comparing the maximum value in the first intermediate result with the second set threshold, thereby avoiding waste of computing power and improving the operation speed of the attention mechanism.

[0119] Optionally or additionally, in some embodiments, the normalization is calculated as an overflow-proof softmax operation. It can be understood that in the attention mechanism, the attention score is used to represent the attention weight of each element in the input sequence to other elements, and according to the block Q of Q i and K's block K j The raw score Logits obtained by performing the inner product operation i,j There may be different value ranges and do not meet the requirements of probability distribution, so a softmax operation can be used to convert the raw scores into attention weights.

[0120] Based on the above description, those skilled in the art can understand that the computing device of the embodiment of the present disclosure can be implemented in a chip, so the present disclosure also discloses a chip, which includes the computing device as described in any of the aforementioned embodiments. In addition, the present disclosure also discloses a board, which includes the aforementioned chip. Finally, the present disclosure discloses a method for performing attention mechanism operations using the computing device of any of the aforementioned embodiments.

[0121] Fig.11 The following is a logical flow chart of the attention mechanism operation according to some embodiments of the present disclosure. The figure fully shows the comparison process of the first set threshold and the second set threshold to better understand the spirit of the present disclosure embodiment. However, those skilled in the art can understand that in other embodiments, the first set threshold and / or the second set threshold may not be set.

[0122] like Fig.11 As shown, first, in step S111, an inner product operation is performed, that is, Logits is calculated. i,j =Q i *K j T Then go to step S112 to perform the operation of subtracting the maximum reference value of the row in the previous round, that is, updating Logits i,j =Logits i,j-last_max. Then enter step S113 to perform the maximum value operation, that is, calculate Max = maxpool (Logits i,j ), where maxpool(x) represents finding the maximum value of the matrix x by row. Then, in step S114, it is determined whether Max is not greater than the second set threshold. If the determination result is "yes", the process proceeds to step S115 to directly set the corresponding output block O i,j All are 0; otherwise, proceed to step S116 to further determine whether Max is not greater than the first set threshold. If the judgment result is "yes", then directly perform the Logits in step S117. i,j Perform softmax operation and calculate P = softmax (Logits i,j ); otherwise, go to step S118 first, and logits i,j Perform an anti-overflow softmax operation and calculate P = softmax (Logits i,j -Max), then enter step S119 to perform the row maximum reference value update operation, set last_max = last_max + Max, this last_max is used as the reference value of the original fraction row maximum value of the subsequent loop rounds. Finally, in step S120, the corresponding output block O is calculated i,j =P*V j .

[0123] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0124] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0125] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0126] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0127] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0128] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A computing device, comprising a control circuit and an operation circuit, wherein: The control circuit is configured to control the operation circuit to perform attention mechanism operation in a block cycle; as well as The operation circuit comprises: A convolution operation circuit, configured to perform the raw score calculation in the attention mechanism operation and the anti-overflow sub-operation in the normalization calculation to obtain a first intermediate result; a normalization calculation circuit configured to perform normalization processing in normalization calculation on the first intermediate result to obtain a second intermediate result; and The matrix multiplication circuit is configured to multiply the second intermediate result with the corresponding value vector V to obtain an output.

2. The computing device according to claim 1, wherein the convolution operation circuit further comprises: a convolution circuit configured to perform a matrix operation of the query vector Q and the key vector K to obtain a raw score; as well as The adding circuit is configured to subtract a reference value of the maximum value of the raw score row in the previous cycle from the raw score to obtain the first intermediate result.

3. The computing device according to claim 2, wherein the normalization computing circuit is further configured to: In response to the maximum value of the first intermediate result not exceeding a first set threshold, directly performing the standardization process on the first intermediate result; and / or In response to the maximum value of the first intermediate result being greater than the first set threshold, the first intermediate result is corrected using the maximum value of the first intermediate result, and the normalization process is performed on the corrected first intermediate result.

4. The computing device according to claim 3, wherein: The operation circuit is further configured to: In response to the maximum value of the first intermediate result being greater than the first set threshold, the reference value of the original fractional row maximum value is updated using the maximum value of the first intermediate result for use by the convolution operation circuit in the next cycle.

5. The computing device according to any one of claims 3 to 4, wherein: The value of the first set threshold is any one of the following: 0, 2, 4.

6. The computing device according to any one of claims 1 to 4, wherein: The operation circuit is further configured to: In response to the maximum value of the first intermediate result not exceeding a second set threshold, the output of the block in the current cycle is set to 0. The computing device according to claim 6 , wherein the second set threshold is determined based on a minimum value of a data type used in the operation.

8. The computing device according to any one of claims 1 to 7, wherein the normalization calculation is an overflow-proof softmax operation.

9. A chip comprising the computing device according to any one of claims 1-8.

10. A board comprising the chip according to claim 9.

11. A method for performing attention mechanism operations using the computing device described in any one of claims 1-8.

Images