Self-attention mechanism calculation method, array, device and system, and storage medium

Abstract

A self-attention mechanism calculation method, array (100), device (200) and system (400), and a computer-readable storage medium (500). The self-attention mechanism calculation array (100) comprises a plurality of multiplier accumulators (10). The self-attention mechanism calculation method comprises: calculating, by means of the plurality of multiplier accumulators (10), an intermediate tensor on the basis of a feature tensor and a weight tensor; and calculating, by means of the plurality of multiplier accumulators (10), a result tensor on the basis of the intermediate tensor.

Description

Self-attention mechanism calculation method, array, device, system and storage medium

[0001] Priority information

[0002] This application claims priority to and the benefit of the filing date of the patent application no. 2024113327138 filed with the China National Intellectual Property Office on September 23, 2024, and incorporates herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, in particular to a self-attention mechanism calculation method, a calculation array, a calculation device, a calculation system and a computer readable storage medium. BACKGROUND

[0004] The Transformer model is another important deep learning algorithm after the Convolutional Neural Network (CNN) and the Recurrent Neural Network (RNN). The self-attention mechanism is the core calculation component of the Transformer model. The Transformer model can not only replace the current CNN and RNN in simple application scenarios, but also complete complex tasks that cannot be achieved by CNN and RNN, while still maintaining high performance. However, the calculation complexity of the self-attention mechanism in the Transformer model is much higher than that of the convolution layer (the core calculation component of CNN) and the long short-term memory network (the core calculation component of RNN). Therefore, how to improve the calculation efficiency of the self-attention mechanism is a problem to be solved. SUMMARY

[0005] The self-attention mechanism calculation method, the calculation array, the calculation device, the calculation system and the computer readable storage medium provided by the embodiments of the present application can solve at least one of the above technical problems.

[0006] The self-attention mechanism calculation method of the embodiments of the present application is applied to a self-attention mechanism calculation array, the self-attention mechanism calculation array includes a plurality of product accumulators, the plurality of product accumulators constitute a systolic array, and the self-attention mechanism calculation method includes:

[0007] An intermediate tensor is calculated by the plurality of product accumulators according to a feature tensor and a weight tensor;

[0008] A result tensor is calculated by the plurality of product accumulators according to the intermediate tensor.

[0009] The self-attention mechanism calculation array of the embodiments of the present application comprises a plurality of product-accumulators, which constitute a systolic array;

[0010] The plurality of product-accumulators are configured to calculate an intermediate tensor according to a feature tensor and a weight tensor;

[0011] The plurality of product-accumulators are further configured to calculate a result tensor according to the intermediate tensor.

[0012] The self-attention mechanism calculation device of the embodiments of the present application comprises the self-attention mechanism calculation array described above.

[0013] The self-attention mechanism calculation system of the embodiments of the present application comprises one or more processors and a memory, and the memory stores a computer program, which, when executed by the processor, implements the self-attention mechanism calculation method described above.

[0014] The computer-readable storage medium of the embodiments of the present application stores a computer program, which, when executed by a processor, implements the self-attention mechanism calculation method described above.

[0015] In the self-attention mechanism calculation method, the calculation array, the calculation device, the calculation system and the computer-readable storage medium of the embodiments of the present application, an intermediate tensor is calculated according to a feature tensor and a weight tensor by a plurality of product-accumulators, and a result tensor is directly calculated based on the calculated intermediate tensor, without removing the intermediate tensor from the self-attention mechanism calculation array. In this way, the frequent access between the intermediate tensor and the external memory is avoided, the calculation efficiency is improved, and the problems of intermediate tensor reuse and storage optimization are solved.

[0016] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter in the description of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:

[0018] FIG. 1 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0019] FIG. 2 is a structural diagram of a self-attention mechanism calculation array according to some embodiments of the present application;

[0020] FIG. 3 is a structural diagram of a product-accumulator according to some embodiments of the present application;

[0021] Figure 4 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0022] Figure 5 is a schematic diagram of data flow within a product accumulator when calculating an intermediate tensor from a feature tensor and a weight tensor according to a self-attention mechanism calculation array according to some embodiments of the present application;

[0023] Figure 6 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0024] Figure 7 is a schematic diagram of data flow of an intermediate tensor in a 4x4 self-attention mechanism calculation array according to some embodiments of the present application;

[0025] Figure 8 is a schematic diagram of matrix multiplication to calculate an intermediate tensor from a feature tensor and a weight tensor according to a self-attention mechanism calculation method according to some embodiments of the present application;

[0026] Figure 9 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0027] Figure 10 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0028] Figure 11 is a schematic diagram of data flow within a product accumulator when calculating a result tensor from an intermediate tensor according to a self-attention mechanism calculation array according to some embodiments of the present application;

[0029] Figure 12 is a schematic diagram of data flow of a sub-result tensor in a 4x4 self-attention mechanism calculation array according to some embodiments of the present application;

[0030] Figure 13 is a schematic diagram of matrix multiplication to calculate a result tensor from an intermediate tensor according to a self-attention mechanism calculation method according to some embodiments of the present application;

[0031] Figure 14 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0032] Figure 15 is a flowchart of a self-attention mechanism calculation method according to some embodiments of the present application;

[0033] Figure 16 is a schematic diagram of the structure of an MxN self-attention mechanism calculation device according to some embodiments of the present application;

[0034] Figure 17 is a schematic diagram of data flow of an intermediate tensor in an NxM self-attention mechanism calculation array according to some embodiments of the present application;

[0035] Figure 18 is a schematic diagram of the modules of a self-attention mechanism calculation system according to some embodiments of the present application;

[0036] FIG. 19 is a schematic diagram of a state of connection of a computer-readable storage medium and a processor according to some embodiments of the present application.

[0037] Legend: self-attention mechanism computing array 100, product accumulator 10, multiplier 11, adder 12, register 13, first selector 141, first input port I1, second input port I2, first output port O1, second selector 142, third input port I3, fourth input port I4, second output port O2, third selector 143, fifth input port I5, sixth input port I6, third output port O3, self-attention mechanism computing device 200, self-attention mechanism computing system 400, processor 410, memory 420, computer-readable storage medium 500, computer program 510, processor 520. DETAILED DESCRIPTION

[0038] The embodiments of the present application are further described below with reference to the drawings. The same or similar reference numerals are used throughout the drawings and the same or similar functions are represented by the same or similar reference numerals. In addition, the embodiments of the present application described below with reference to the drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as a limitation of the present application.

[0039] The Transformer model is another important deep learning algorithm after CNN and RNN. Unlike CNN and RNN, which can only focus on a specific field, for example, CNN is mainly applied to the field of computer vision, and RNN is mainly applied to the field of natural language processing, the Transformer model has very good performance in the above two different real scenes. This is mainly due to the core component of the Transformer model, the self-attention mechanism Self-Attention.

[0040] The Transformer model not only can replace the existing CNN and RNN in simple application scenarios, but also can complete complex tasks that CNN and RNN cannot achieve, and still maintains high performance. The basic structure of the large language model currently widely discussed is the Transformer model. However, the cost of the Transformer model with such high performance is that the computational complexity of the self-attention mechanism is much higher than that of the convolutional layer and the long short-term memory network.

[0041] With the increasing application of deep learning algorithms, in mobile chips, a special deep learning accelerator Neural Processing Unit, NPU is designed to improve the computing speed and energy efficiency of deep learning algorithms in mobile devices. However, the current NPU focuses on the optimization of convolution calculation and small-scale matrix multiplication, and the optimization of self-attention mechanism is relatively limited.

[0042] In related technologies, the accelerator structure A3 for self-attention mechanism mainly consists of three parts, including Dot-Product module, Exponent Computation module and Output Computation module. Among them, the Dot-Product module is responsible for the matrix multiplication operation before the normalization exponential function softmax operation, the Exponent Computation module is responsible for the calculation of softmax, and the Output Computation module is responsible for the last step of matrix multiplication operation.

[0043] The main problem of A3 is that the matrix multiplication operation in the self-attention mechanism is decomposed into matrix-vector multiplication operation, which limits the data reuse rate and makes it difficult to deploy A3 on hardware with large-scale computing resources. In addition, the bottleneck of the current transformer model accelerator performance is mainly in the bandwidth, and the structure proposed by A3 cannot solve this problem.

[0044] In other related technologies, an accelerator SOCC20 is proposed to optimize the hardware of self-attention mechanism and Feed-Forward Network, FFN at the same time. The accelerator adopts a depth-first acceleration calculation strategy between adjacent self-attention mechanism and FFN modules, which can effectively reduce the movement between computing resources and memory. However, the portability of this structure is poor, and only when the input sequence length of the Transformer model is less than or equal to a certain length, the performance will be better. But in actual hardware design, the input sequence length will be greatly limited by hardware resources. In actual application scenarios, especially in natural language processing related applications, the input sequence length of the Transformer model has strong randomness. Therefore, in complex and variable Transformer models, SOCC20 still has strong limitations.

[0045] In the above two schemes, the reuse and storage optimization of intermediate results during self-attention mechanism calculation are not considered. The memory bandwidth of intermediate results is the key to limit the computing efficiency of the accelerator.

[0046] Therefore, referring to FIG. 1 and FIG. 2, the self-attention mechanism computing method provided by the embodiments of the present application is applied to a self-attention mechanism computing array 100. The self-attention mechanism computing array 100 includes a plurality of multiply-accumulate units 10, which form a systolic array. The self-attention mechanism computing method includes:

[0047] 010: calculating an intermediate tensor by the plurality of multiply-accumulate units 10 according to the feature tensor and the weight tensor;

[0048] 020: calculating a result tensor by the plurality of multiply-accumulate units 10 according to the intermediate tensor.

[0049] In the self-attention mechanism computing method of the embodiments of the present application, the intermediate tensor is calculated by the plurality of multiply-accumulate units 10 according to the feature tensor and the weight tensor, and the result tensor is directly calculated based on the calculated intermediate tensor, without removing the intermediate tensor from the self-attention mechanism computing array 100. In this way, the frequent access between the intermediate tensor and the external memory is avoided, the computing efficiency is improved, and the problems of intermediate tensor reuse and storage optimization are solved.

[0050] Specifically, the self-attention mechanism computing array 100 includes a plurality of multiply-accumulate (MAC) units 10, which are arranged in a matrix form, and each row of two adjacent multiply-accumulate units 10 are connected to each other, and each column of two adjacent multiply-accumulate units 10 are connected to each other, forming a systolic array.

[0051] The self-attention mechanism computing array 100 can be flexibly configured to different specifications according to actual computing power requirements. For example, the self-attention mechanism computing array 100 can include 4x4 multiply-accumulate units 10. The embodiments of the present application are described by taking the self-attention mechanism computing array 100 including 4x4 multiply-accumulate units 10 as an example.

[0052] The feature tensor is represented by X, and the weight tensor is represented by W q / W k The feature tensor X and the weight tensor W q / W k The self-attention mechanism computing array 100 is input, and the plurality of multiply-accumulate units 10 can calculate the intermediate tensor Q / K according to the feature tensor X and the weight tensor W q / W k . After that, the plurality of multiply-accumulate units 10 can directly calculate the result tensor QxK according to the intermediate tensor Q / K T .

[0053] Referring to Figures 2 to 5, in some embodiments, each multiply-accumulate unit 10 includes a multiplier 11, a first selector 141, and a second selector 142. Both the first selector 141 and the second selector 142 are connected to the multiplier 11. The weight tensor includes a first weight tensor, and the intermediate tensor includes a first intermediate tensor. The intermediate tensor (i.e., 010) is calculated by multiple multiply-accumulate units 10 based on the feature tensor and the weight tensor, including:

[0054] 011: Select the feature tensor through the first selector 141 and input it into the multiplier 11;

[0055] 012: The first weight tensor is selected by the second selector 142 and input to the multiplier 11;

[0056] 013: The first intermediate tensor is obtained by multiplying the feature tensor with the first weight tensor through multiplier 11.

[0057] Specifically, as shown in Figure 3, each multiply-accumulate unit 10 includes a multiplier 11, a first selector 141, and a second selector 142, both of which are connected to the multiplier 11. Both the first selector 141 and the second selector 142 have a two-input, one-output structure. The first selector 141 includes a first input port I1, a second input port I2, and a first output port O1, and is connected to the multiplier 11 through the first output port O1. The second selector 142 includes a third input port I3, a fourth input port I4, and a second output port O2, and is connected to the multiplier 11 through the second output port O2.

[0058] Referring to Figures 3 and 5, the first input port I1 of the first selector 141 is used to input the feature tensor, and the third input port I3 of the second selector 142 is used to input the weight tensor W. q / W k Weight tensor W q / W k Including the first weight tensor, with W q This indicates that the intermediate tensor Q / K includes the first intermediate tensor, denoted by Q.

[0059] For at least a portion of the multiple multiply-accumulate units 10, the first output port O1 of the first selector 141 selects the feature tensor X as input to the multiplier 11, and the second output port O2 of the second selector 142 selects the first weight tensor Wq as input to the multiplier 11. The multiplier 11 then multiplies the feature tensor X with the first weight tensor Wq. q Multiplying them together yields the first intermediate tensor Q, which means X × W is completed within the multiply-accumulate unit 10. q = The calculation of Q corresponds to the first step in the sub-attention mechanism.

[0060] Referring to FIG. 2 to FIG. 5, in some embodiments, each product-accumulator 10 further comprises a register 13 connected to the multiplier 11, and the self-attention mechanism calculation method further comprises:

[0061] 014: storing the first intermediate tensor by the register 13.

[0062] Specifically, the register 13 is connected to the multiplier 11, and after the multiplier 11 calculates the first intermediate tensor Q, the first intermediate tensor Q is input to the register 13 for storage, so as to facilitate subsequent calculation.

[0063] Referring to FIG. 2, FIG. 3, FIG. 5 and FIG. 6, in some embodiments, the weight tensor further comprises a second weight tensor, and the intermediate tensor further comprises a second intermediate tensor. The intermediate tensor (i.e. 010) calculated by the plurality of product-accumulators 10 according to the feature tensor and the weight tensor further comprises:

[0064] 015: selecting the second weight tensor to be input to the multiplier 11 by the second selector 142;

[0065] 016: multiplying the feature tensor and the second weight tensor to obtain the second intermediate tensor by the multiplier 11.

[0066] Specifically, in combination with FIG. 3 and FIG. 5, the first input port I1 of the first selector 141 is used to input the feature tensor, and the third input port I3 of the second selector 142 is used to input the weight tensor W q / W k . The weight tensor W q / W k comprises a second weight tensor, denoted as W k , and the intermediate tensor Q / K comprises a second intermediate tensor, denoted as K.

[0067] For at least part of the plurality of product-accumulators 10, the first output port O1 of the first selector 141 selects the feature tensor X to be input to the multiplier 11, and the second output port O2 of the second selector 142 selects the second weight tensor W k to be input to the multiplier 11, and the multiplier 11 multiplies the feature tensor X and the second weight tensor W k to obtain the second intermediate tensor K, that is, the calculation of X×W k = K is completed in the product-accumulator 10, corresponding to the calculation of the first step in the sub-attention mechanism.

[0068] Referring to FIG. 2, FIG. 3, FIG. 5 and FIG. 6, in some embodiments, each product-accumulator 10 further comprises a register 13 connected to the multiplier 11, and the self-attention mechanism calculation method further comprises:

[0069] 017: store the second intermediate tensor via register 13.

[0070] Specifically, register 13 is connected with multiplier 11, after multiplier 11 calculates the second intermediate tensor K, the second intermediate tensor K is input to register 13 for storage, so as to facilitate subsequent calculation.

[0071] Referring to FIG. 2, FIG. 3 and FIG. 5, in some embodiments, the first weight tensor is selected by the second selector 142 to be input to the multiplier 11 (i.e. 012), including:

[0072] For the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the first weight tensor is selected by the second selector 142 to be input to the multiplier 11;

[0073] The feature tensor is multiplied by the first weight tensor via the multiplier 11 to obtain the first intermediate tensor (i.e. 013), including:

[0074] For the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the feature tensor is multiplied by the first weight tensor via the multiplier 11 to obtain the first intermediate tensor;

[0075] Wherein, M is an odd number, M≥1.

[0076] Specifically, M is an odd number, M≥1, then (M+1) is an even number. For the plurality of product accumulators 10 of the (M+1)th row (i.e. even row) of the systolic array, the first weight tensor W q is selected by the second selector 142 to be input to the multiplier 11. It can be understood that the first weight tensor W q includes a plurality of first weight values, and the plurality of first weight values are respectively input to the multipliers 11 of the plurality of product accumulators 10 of the (M+1)th row, and each first weight value is correspondingly input to one multiplier 11.

[0077] For example, for the systolic array as shown in FIG. 2, the second row and the fourth row each include 4 product accumulators 10, the first weight tensor W q is selected by the second selector 142 to be input to the multiplier 11, and the first weight tensor W q includes first weight values W q0 , W q1 , W q2 , W q4 , the four first weight values are first input to the multipliers 11 of the 4 product accumulators 10 of the second row, and then transmitted to the multipliers 11 of the 4 product accumulators 10 of the fourth row.

[0078] It should be noted that the feature tensor X includes a plurality of feature values, which can be respectively input to the multipliers 11 of the plurality of product accumulators 10 in different even rows. For example, the feature tensor X includes feature values X0, X1, for the systolic array as shown in FIG. 2, the feature value X0 is input to the multipliers 11 of the 4 product accumulators 10 in the second row, and the feature value X1 is input to the multipliers 11 of the 4 product accumulators 10 in the fourth row.

[0079] Then, the plurality of product accumulators 10 in the (M+1)th row of the systolic array can multiply the feature tensor X and the first weight tensor W q to obtain a first intermediate tensor Q. For the 4 product accumulators 10 in the second row of the systolic array in FIG. 2, the feature value X0 is respectively multiplied by the first weight values W q0 , W q1 , W q2 , and W q4 through the multipliers 11 to obtain Q 00 , Q 01 , Q 02 , and Q 03 . For the 4 product accumulators 10 in the fourth row of the systolic array in FIG. 2, the feature value X1 is respectively multiplied by the first weight values W q0 , W q1 , W q2 , and W q4 through the multipliers 11 to obtain Q 10 , Q 11 , Q 12 , and Q 13 . The results of the second row and the fourth row together constitute the first intermediate tensor Q, and the calculation result is shown in FIG. 7, and the corresponding matrix multiplication schematic diagram is shown in FIG. 8.

[0080] Referring to FIGS. 2, 3, and 5, in some embodiments, the second weight tensor is selected by the second selector 142 to be input to the multipliers 11 (i.e., 015), including:

[0081] For the plurality of product accumulators 10 in the Mth row of the systolic array, the second weight tensor is selected by the second selector 142 to be input to the multipliers 11;

[0082] The feature tensor is multiplied by the second weight tensor through the multipliers 11 to obtain a second intermediate tensor (i.e., 016), including:

[0083] For the plurality of product accumulators 10 in the Mth row of the systolic array, the feature tensor is multiplied by the second weight tensor through the multipliers 11 to obtain a second intermediate tensor;

[0084] Wherein, M is an odd number, and M≥1.

[0085] In particular, M is an odd number, M≥1. For the plurality of product accumulators 10 in the Mth row (i.e. the odd row) of the systolic array, the second weight tensor W k is input to the multipliers 11. It can be appreciated that the second weight tensor W k comprises a plurality of second weight values, the plurality of second weight values being input to the multipliers 11 of the plurality of product accumulators 10 in the Mth row respectively, each second weight value being input to one multiplier 11 correspondingly.

[0086] For example, for the systolic array as shown in Fig. 2, the first and third rows each comprise 4 product accumulators 10, the second weight tensor W k is input to the multipliers 11, the second weight tensor W k comprises second weight values W k0 , W k1 , W k2 , W k4 , the four second weight values being input to the multipliers 11 of the 4 product accumulators 10 in the first row first and then to the multipliers 11 of the 4 product accumulators 10 in the third row.

[0087] It is to be noted that the feature tensor X comprises a plurality of feature values, the plurality of feature values can be input to the multipliers 11 of the plurality of product accumulators 10 in different odd rows respectively. For example, the feature tensor X comprises feature values X0, X1, for the systolic array as shown in Fig. 2, the feature value X0 is input to the multipliers 11 of the 4 product accumulators 10 in the first row, and the feature value X1 is input to the multipliers 11 of the 4 product accumulators 10 in the third row.

[0088] Then the plurality of product accumulators 10 in the Mth row of the systolic array can obtain a second intermediate tensor K by multiplying the feature tensor X and the second weight tensor W k via the multipliers 11. For the 4 product accumulators 10 in the first row of the systolic array in Fig. 2, the feature value X0 is multiplied by the second weight values W k0 , W k1 , W k2 , W k4 via the multipliers 11 respectively to obtain K 00 , K 01 , K 02 , K 03 . For the 4 product accumulators 10 in the third row of the systolic array in Fig. 2, the feature value X1 is multiplied by the second weight values W k0 , W k1 , W k2 , W k4 via the multipliers 11 respectively to obtain K 10 , K 11 , K 12 , K13 The results of the first row and the third row jointly constitute a second intermediate tensor K, and the calculation result is shown in FIG. 7, and the corresponding matrix multiplication schematic diagram is shown in FIG. 8.

[0089] Referring to FIG. 2 and FIG. 9, in some embodiments, the feature tensor includes a plurality of channel sub-feature tensors, and the weight tensor includes a plurality of channel sub-weight tensors. The intermediate tensor (010) is calculated by the plurality of product accumulators 10 according to the feature tensor and the weight tensor, including:

[0090] 018: The plurality of product accumulators 10 respectively calculate a plurality of sub-intermediate tensors according to the plurality of channel sub-feature tensors and the plurality of channel sub-weight tensors at a plurality of time points.

[0091] 019: The plurality of product accumulators 10 accumulate the plurality of sub-intermediate tensors to obtain the intermediate tensor.

[0092] Specifically, the feature tensor X includes a plurality of channel sub-feature tensors, and the weight tensor W q / W k includes a plurality of channel sub-weight tensors corresponding thereto, and each channel sub-feature tensor is calculated with the corresponding sub-weight tensor, and the plurality of channel calculations are unfolded in the time dimension. That is, the plurality of product accumulators 10 respectively calculate a plurality of sub-intermediate tensors according to the plurality of channel sub-feature tensors and the plurality of channel sub-weight tensors at a plurality of time points. The plurality of product accumulators 10 can accumulate the plurality of sub-intermediate tensors to obtain the intermediate tensor Q / K.

[0093] Taking an example that the feature tensor includes two channel sub-feature tensors and the weight tensor includes two channel sub-weight tensors, at the first time point, in the plurality of product accumulators 10, the first selector 141 selects the first sub-feature tensor X 1 input to the multiplier 11, and the second selector 142 selects the first sub-weight tensor W q 1 / W k 1 input to the multiplier 11. The multiplier 11 calculates the first sub-feature tensor X 1 and the first sub-weight tensor W q 1 / W k 1 to obtain the first sub-intermediate tensor Q 1 / K 1 , and stores the first sub-intermediate tensor Q 1 / K 1 in the register 13.

[0094] In some embodiments, each product accumulator 10 further comprises a third selector 143, the third selector 143, the multiplier 11 and the register 13 are all connected with the adder 12. The third selector 143 comprises a fifth input port I5, a sixth input port I6 and a third output port O3, the third selector 143 is connected with the adder 12 through the third output port O3, and the third selector 143 is connected with the register 13 in the same product accumulator 10 through the sixth input port I6.

[0095] At the second time, among the plurality of product accumulators 10, the first selector 141 selects the second sub-feature tensor X 2 input to the multiplier 11, and the second selector 142 selects the second sub-weight tensor W q 2 / W k 2 input to the multiplier 11. The multiplier 11 calculates the second sub-feature tensor X 2 and the second sub-weight tensor W q 2 / W k 2 to obtain the second sub-intermediate tensor Q 2 / K 2 At this time, the third selector 143 can select the first sub-intermediate tensor Q 1 / K 1 stored in the register 13 and output to the adder 12. The adder 12 adds the first sub-intermediate tensor Q 1 / K 1 and the second sub-intermediate tensor Q 2 / K 2 and inputs to the register 13 for storage to be added with the intermediate tensor calculated at the next time. In this way, the plurality of product accumulators 10 add the plurality of sub-intermediate tensors to obtain the intermediate tensor Q / K.

[0096] Please refer to FIG. 2, FIG. 10 and FIG. 11. In some embodiments, the register 13 is further connected with the second selector 142. The result tensor (i.e. 020) calculated by the plurality of product accumulators 10 according to the intermediate tensor comprises:

[0097] 021: input the first intermediate tensor to the multiplier 11 through the first selector 141;

[0098] 022: input the second intermediate tensor to the multiplier 11 by selecting from the register 13 through the second selector 142;

[0099] 023: multiply the first intermediate tensor and the second intermediate tensor by the multiplier 11 to obtain a sub-result tensor;

[0100] 024: calculating the result tensor according to the plurality of sub-result tensors corresponding to the plurality of product accumulators 10.

[0101] Specifically, in combination with FIG. 2, FIG. 3 and FIG. 11, for the plurality of product accumulators 10, the register 13 of one product accumulator 10 can be connected with the first selector 141 of the product accumulator 10 adjacent to the same column, specifically can be connected with the second input port I2 of the first selector 141, so that when the register 13 of the product accumulator 10 stores the second intermediate tensor K calculated in the foregoing embodiments, the first intermediate tensor Q from the adjacent row can be selected by the first selector 141 and input to the multiplier 11.

[0102] Inside one product accumulator 10, the register 13 is connected with the second selector 142, specifically can be connected with the fourth input port I4 of the second selector 142, so that the second intermediate tensor K can be selected by the second selector 142 from the register 13 and input to the multiplier 11.

[0103] The sub-result tensor can be obtained by multiplying the first intermediate tensor Q and the second intermediate tensor K by the multiplier 11, and the sub-result tensor is shown in FIG. 12. The result tensor QxK can be calculated according to the plurality of sub-result tensors corresponding to the plurality of product accumulators 10. T The schematic diagram of the corresponding matrix multiplication is shown in FIG. 14, that is, the calculation corresponding to the second step in the self-attention mechanism.

[0104] Please refer to FIG. 2, FIG. 11 and FIG. 13, in some embodiments, the register 13 is also connected with the second selector 142, and the result tensor is calculated by the plurality of product accumulators 10 according to the intermediate tensor, including:

[0105] 025: selecting the second intermediate tensor by the first selector 141 and inputting to the multiplier 11;

[0106] 026: selecting the first intermediate tensor from the register 13 by the second selector 142 and inputting to the multiplier 11;

[0107] 027: multiplying the first intermediate tensor and the second intermediate tensor by the multiplier 11 to obtain the sub-result tensor;

[0108] 028: calculating the result tensor according to the plurality of sub-result tensors corresponding to the plurality of product accumulators 10.

[0109] Specifically, in combination with FIG. 2, FIG. 3 and FIG. 11, for the plurality of product accumulators 10, the register 13 of one product accumulator 10 can be connected with the first selector 141 of the product accumulator 10 adjacent to the same column, specifically can be connected with the second input port I2 of the first selector 141, so that when the register 13 of the product accumulator 10 stores the first intermediate tensor Q calculated in the foregoing embodiment, the second intermediate tensor K from the adjacent row can be selected by the first selector 141 to be input to the multiplier 11.

[0110] Inside the product accumulator 10, the register 13 is connected with the second selector 142, specifically can be connected with the fourth input port I4 of the second selector 142, so that the first intermediate tensor Q can be selected by the second selector 142 from the register 13 to be input to the multiplier 11.

[0111] The first intermediate tensor Q can be multiplied with the second intermediate tensor K by the multiplier 11 to obtain a sub-result tensor, and the sub-result tensor is shown in FIG. 12. The result tensor Q x K can be calculated according to the plurality of sub-result tensors corresponding to the plurality of product accumulators 10. T The schematic diagram of the corresponding matrix multiplication is shown in FIG. 14, that is, the calculation corresponding to the second step in the self-attention mechanism.

[0112] Please refer to FIG. 2 and FIG. 11, in some embodiments, the first selector 141 of the Mth row of the systolic array is connected with the register 13 of the (M+1)th row of the systolic array, and the first intermediate tensor is selected by the first selector 141 to be input to the multiplier 11 (i.e. 021), including:

[0113] For the plurality of product accumulators 10 of the Mth row of the systolic array, the first intermediate tensor from the (M+1)th row of the systolic array is selected by the first selector 141 to be input to the multiplier 11;

[0114] The second intermediate tensor is selected by the second selector 142 from the register 13 to be input to the multiplier 11 (i.e. 022), including:

[0115] For the plurality of product accumulators 10 of the Mth row of the systolic array, the second intermediate tensor is selected by the second selector 142 from the register 13 to be input to the multiplier 11;

[0116] Wherein, M is an odd number, M≥1.

[0117] In particular, the first selector 141 of the Mth row of the systolic array is connected with the register 13 of the (M+1)th row of the systolic array, M is an odd number, M≥1, M can be 1, 3, 5, 7 or any other odd number greater than 1, and (M+1) can be 2, 4, 6, 8 or any other even number greater than 2. For the systolic array in FIG. 2, the first selectors 141 of the four product accumulators 10 in the first row are connected with the registers 13 of the four product accumulators 10 in the second row, and the first selectors 141 of the four product accumulators 10 in the third row are connected with the registers 13 of the four product accumulators 10 in the fourth row.

[0118] For the plurality of product accumulators 10 in the Mth row of the systolic array, the register 13 stores the second intermediate tensor K calculated by the foregoing embodiment, and the first selector 141 can select the first intermediate tensor Q from the (M+1)th row of the systolic array to input to the multiplier 11.

[0119] For the plurality of product accumulators 10 in the Mth row of the systolic array, the register 13 stores the second intermediate tensor K calculated by the foregoing embodiment, and the second selector 142 can select the second intermediate tensor K from the register 13 to input to the multiplier 11, so that the multiplier can multiply the first intermediate tensor Q to obtain a sub-result tensor.

[0120] For example, as shown in FIG. 7, in the four product accumulators 10 in the first row of the systolic array, the register 13 stores the second intermediate tensor K (K 00 , K 01 , K 02 , K 03 ) calculated by the foregoing embodiment, and the first selector 141 can select the first intermediate tensor Q (Q 00 , Q 01 , Q 02 , Q 03 ) from the second row of the systolic array to input to the multiplier 11. The multiplier 11 can multiply the first intermediate tensor Q and the second intermediate tensor K to obtain a sub-result tensor. As shown in FIG. 12, the four product accumulators 10 in the first row of the systolic array correspond to the sub-result tensors Q 00 K 00 , Q 01 K 01 , Q 02 K 02 , Q 03 K 03 , respectively.

[0121] In the four product accumulators 10 in the third row of the systolic array, the register 13 stores the second intermediate tensor K (K 10 , K 11 , K12 13 ), the first selector 141 can select the first intermediate tensor Q(Q 10 11 12 13 ) to the multiplier 11. Through the multiplier 11, the first intermediate tensor Q can be multiplied with the second intermediate tensor K to obtain a sub-result tensor. As shown in FIG. 12, the 4 product accumulators 10 of the third row of the systolic array correspond to the sub-result tensors Q 10 10 11 11 12 12 13 13 .

[0122] Referring to FIG. 2 and FIG. 11, in some embodiments, the first selector 141 of the (M+1)th row of the systolic array is connected with the register 13 of the (M+2)th row of the systolic array. The second intermediate tensor is selected by the first selector 141 to be input to the multiplier 11 (i.e. 025), including:

[0123] For the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the second intermediate tensor from the (M+2)th row of the systolic array is selected by the first selector 141 to be input to the multiplier 11;

[0124] The first intermediate tensor is selected by the second selector 142 from the register 13 to be input to the multiplier 11 (i.e. 026), including:

[0125] For the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the first intermediate tensor is selected by the second selector 142 from the register 13 to be input to the multiplier 11;

[0126] Wherein, M is an odd number, M≥1.

[0127] Specifically, the first selector 141 of the (M+1)th row of the systolic array is connected with the register 13 of the (M+2)th row of the systolic array, M is an odd number, M≥1, M can be 1, 3, 5, 7 or other arbitrary odd number greater than 1, then (M+2) can be 3, 5, 7, 9 or other arbitrary odd number greater than 3. Then for the systolic array in FIG. 2, the first selectors 141 of the 4 product accumulators 10 of the second row are connected with the registers 13 of the 4 product accumulators 10 of the third row.

[0128] ​​​​​​​​​​​For the plurality of product accumulators 10 of the (M+1)th row of systolic array, the register 13 stores the first intermediate tensor Q calculated by the foregoing embodiment, and the first selector 141 can select the second intermediate tensor K from the (M+2)th row of systolic array to input to the multiplier 11.

[0129] For the plurality of product accumulators 10 of the (M+2)th row of systolic array, the register 13 stores the first intermediate tensor Q calculated by the foregoing embodiment, and the second selector 142 can select the first intermediate tensor Q from the register 13 to input to the multiplier 11, so that the multiplier can multiply the first intermediate tensor Q with the second intermediate tensor K to obtain a sub-result tensor.

[0130] For example, as shown in FIG. 7, in the 4 product accumulators 10 of the second row of systolic array, the register 13 stores the first intermediate tensor Q (Q 00 , Q 01 , Q 02 , Q 03 ) calculated by the foregoing embodiment, and the first selector 141 can select the second intermediate tensor K (K 10 , K 11 , K 12 , K 13 ) from the third row of systolic array to input to the multiplier 11. The multiplier 11 can multiply the first intermediate tensor Q with the second intermediate tensor K to obtain a sub-result tensor. As shown in FIG. 12, the 4 product accumulators 10 of the second row of systolic array correspond to the sub-result tensors Q 00 K 10 , Q 01 K 11 , Q 02 K 12 , Q 03 K 13 , respectively.

[0131] Please refer to FIG. 2 and FIG. 11. In some embodiments, the first selector 141 of the last row of systolic array is connected with the register 13 of the first row of systolic array, and the first selector 141 selects the second intermediate tensor to input to the multiplier 11 (i.e. 025), including:

[0132] For the plurality of product accumulators 10 of the last row of systolic array, the first selector 141 selects the second intermediate tensor from the first row of systolic array to input to the multiplier 11;

[0133] The second selector 142 selects the first intermediate tensor from the register 13 to input to the multiplier 11 (i.e. 026), including:

[0134] For the plurality of product accumulators 10 of the last row of systolic array, the first intermediate tensor is selected from the register 13 by the second selector 142 to the multiplier 11;

[0135] wherein M is an odd number, M≥1.

[0136] In particular, the first selector 141 of the last row of systolic array is connected with the register 13 of the first row of systolic array. Then for the systolic array in FIG. 2, the first selectors 141 of the 4 product accumulators 10 of the fourth row are connected with the registers 13 of the 4 product accumulators 10 of the first row respectively.

[0137] For the plurality of product accumulators 10 of the last row of systolic array, the register 13 stores the first intermediate tensor Q calculated by the foregoing embodiment, and the second intermediate tensor K from the first row of systolic array can be selected by the first selector 141 to the multiplier 11.

[0138] For the plurality of product accumulators 10 of the first row of systolic array, the register 13 stores the first intermediate tensor Q calculated by the foregoing embodiment, and the first intermediate tensor Q can be selected from the register 13 by the second selector 142 to the multiplier 11, so that the multiplier can multiply the first intermediate tensor Q with the second intermediate tensor K to obtain a sub-result tensor.

[0139] For example, as shown in FIG. 7, in the 4 product accumulators 10 of the fourth row of systolic array, the register 13 stores the first intermediate tensor Q (Q 10 , Q 11 , Q 12 , Q 13 ) calculated by the foregoing embodiment, and the second intermediate tensor K (K 00 , K 01 , K 02 , K 03 ) from the first row of systolic array can be selected by the first selector 141 to the multiplier 11. The first intermediate tensor Q can be multiplied with the second intermediate tensor K by the multiplier 11 to obtain a sub-result tensor. As shown in FIG. 12, the sub-result tensors corresponding to the 4 product accumulators 10 of the fourth row of systolic array are Q 10 K 00 , Q 11 K 01 , Q 12 K 02 , Q 13 K 03 respectively.

[0140] Referring to FIG. 2 and FIG. 15, in some embodiments, the result tensor comprises a plurality of output result tensors of multiple rows. The result tensor is calculated according to a plurality of sub-result tensors corresponding to a plurality of product accumulators 10 of each row of the systolic array (i.e. 024 and 028), comprising:

[0141] 029: accumulating the plurality of sub-result tensors corresponding to the plurality of product accumulators 10 of each row of the systolic array to obtain the output result tensor of the row.

[0142] Specifically, the result tensor comprises a plurality of output result tensors of multiple rows. The output result tensor of each row is obtained by accumulating the plurality of sub-result tensors corresponding to the plurality of product accumulators 10 of each row of the systolic array, and the result tensor can be constructed according to the output result tensors.

[0143] For example, as shown in FIG. 12, the result tensor comprises output results of four rows of the systolic array, and the output result tensor of each row corresponds to an element in the result tensor. The corresponding matrix multiplication schematic diagram is shown in FIG. 13. The output result tensor of the first row of the systolic array corresponds to the element at position (1, 1) in Q x K T of FIG. 13, the output result tensor of the second row of the systolic array corresponds to the element at position (2, 1) in Q x K T of FIG. 13, the output result tensor of the third row of the systolic array corresponds to the element at position (2, 2) in Q x K T of FIG. 13, and the output result tensor of the fourth row of the systolic array corresponds to the element at position (1, 2) in Q x K T of FIG. 13.

[0144] The accumulation of the plurality of sub-result tensors corresponding to the plurality of product accumulators 10 of each row of the systolic array is described in detail below.

[0145] Referring to FIG. 2, FIG. 3 and FIG. 11, in some embodiments, the output result tensor of each row of the systolic array is obtained by accumulating the plurality of sub-result tensors corresponding to the plurality of product accumulators 10 of each row of the systolic array (i.e. 029), comprising:

[0146] For the plurality of product accumulators 10 of the first column of the systolic array, the sub-result tensor of the first column of the systolic array is input to the adder 12 through the multiplier 11;

[0147] For the plurality of product accumulators 10 of the first column of the systolic array, the sub-result tensor of the first column of the systolic array is output through the register 13.

[0148] Specifically, the multiple product-accumulators in each column of the systolic array perform the same operation when accumulating the multiple sub-result tensors corresponding to the multiple product-accumulators in each row of the systolic array. Each product-accumulator 10 further comprises an adder 12, and the multiplier 11, the adder 12 and the register 13 are connected in sequence in each product-accumulator 10. For the multiple product-accumulators in the first column of the systolic array, the sub-result tensors are calculated by the multipliers 11 respectively, and then input to the adders 12, and then input to the registers 13 via the adders 12, and finally output the sub-result tensors of the first column of the systolic array via the registers 13.

[0149] Referring to FIGS. 2, 3 and 11, in some embodiments, the third selectors 143 in the second column of the systolic array are further connected with the registers 13 in the first column of the systolic array, and accumulate the multiple sub-result tensors corresponding to the multiple product-accumulators 10 in each row of the systolic array to obtain the output result tensors of the row, and further comprise:

[0150] For the multiple product-accumulators 10 in the second column of the systolic array, the sub-result tensors from the first column of the systolic array are selected by the third selectors 143 and input to the adders 12;

[0151] For the multiple product-accumulators 10 in the second column of the systolic array, the sub-result tensors of the second column of the systolic array are input to the adders 12 by the multipliers 11;

[0152] For the multiple product-accumulators 10 in the second column of the systolic array, the sub-result tensors of the first column of the systolic array and the sub-result tensors of the second column of the systolic array are added by the adders 12 to obtain the accumulated result tensors of the second column of the systolic array;

[0153] For the multiple product-accumulators 10 in the second column of the systolic array, the accumulated result tensors of the second column of the systolic array are output by the registers 13.

[0154] Specifically, each product-accumulator 10 further comprises a third selector 143, and the third selector 143, the multiplier 11 and the register 13 are connected with the adder 12. The third selector 143 comprises a fifth input port I5, a sixth input port I6 and a third output port O3, and the third selector 143 is connected with the adder 12 through the third output port O3, and the third selector 143 is connected with the register 13 in the same product-accumulator 10 through the sixth input port I6. In addition, the third selectors 143 in the second column of the systolic array are further connected with the registers 13 in the first column of the systolic array through the fifth input ports I5.

[0155] For the plurality of product accumulators 10 in the second column of systolic array, the accumulated result tensor from the first column of systolic array is selected by the third selector 143 to be input to the adder 12. After the multiplier 11 of the plurality of product accumulators 10 in the second column of systolic array calculates the sub-result tensor, the sub-result tensor is input to the adder 12. The adder 12 of the plurality of product accumulators 10 in the second column of systolic array receives the sub-result tensor of the second column of systolic array and the sub-result tensor output from the first column of systolic array, and adds them to obtain the accumulated result tensor of the second column of systolic array.

[0156] For example, as shown in FIG. 12, the accumulated result tensors of the second column of systolic array are sequentially Q 00 K 00 +Q 01 K 01 , Q 00 K 10 +Q 01 K 11 , Q 10 K 10 +Q 11 K 11 , Q 10 K 00 +Q 11 K 01 .

[0157] Referring to FIGS. 2, 3 and 11, in some embodiments, the third selector 143 of the (N+1)th column of systolic array is also connected with the register 13 of the Nth column of systolic array, and the plurality of sub-result tensors corresponding to the plurality of product accumulators 10 in each row of systolic array are added to obtain the output result tensor of the row (i.e., 029), including:

[0158] For the plurality of product accumulators 10 in the (N+1)th column of systolic array, the accumulated result tensor from the Nth column of systolic array is selected by the third selector 143 to be input to the adder 12;

[0159] For the plurality of product accumulators 10 in the (N+1)th column of systolic array, the sub-result tensor of the (N+1)th column of systolic array is input to the adder 12 by the multiplier 11;

[0160] For the plurality of product accumulators 10 in the (N+1)th column of systolic array, the accumulated result tensor of the Nth column of systolic array and the sub-result tensor of the (N+1)th column of systolic array are added by the adder 12 to obtain the accumulated result tensor of the (N+1)th column of systolic array;

[0161] For the plurality of product accumulators 10 in the last column of systolic array, the accumulated result tensor of the last column of systolic array is output by the register 13 as the output result tensor of the row;

[0162] wherein N is an integer, N≥2.

[0163] Specifically, the third selector 143 of the (N+1)th column of systolic array is further connected with the register 13 of the Nth column of systolic array through the fifth input port I5. N is an integer, N≥2, and N can be 3, 4, 5, 6 or any other integer greater than 2. For example, when N=2, the third selector 143 of the third column of systolic array is further connected with the register 13 of the second column of systolic array through the fifth input port I5; when N=3, the third selector 143 of the fourth column of systolic array is further connected with the register 13 of the third column of systolic array through the fifth input port I5; and so on, which will not be described one by one here.

[0164] For the plurality of product accumulators 10 of the (N+1)th column of systolic array, the accumulated result tensor from the Nth column of systolic array can be selected by the third selector 143 to be input to the adder 12. After the multiplier 11 of the plurality of product accumulators 10 of the (N+1)th column of systolic array calculates a sub-result tensor, the sub-result tensor can be input to the adder 12. The adder 12 of the plurality of product accumulators 10 of the (N+1)th column of systolic array receives the sub-result tensor output by the (N+1)th column of systolic array and the accumulated result tensor of the Nth column of systolic array, and adds them to obtain the accumulated result tensor of the (N+1)th column of systolic array.

[0165] For example, for the plurality of product accumulators 10 of the third column of systolic array, the accumulated result tensor from the second column of systolic array can be selected by the third selector 143 to be input to the adder 12. After the multiplier 11 of the plurality of product accumulators 10 of the third column of systolic array calculates a sub-result tensor, the sub-result tensor can be input to the adder 12. The adder 12 of the plurality of product accumulators 10 of the third column of systolic array receives the sub-result tensor of the third column of systolic array and the accumulated result tensor output by the second column of systolic array, and adds them to obtain the accumulated result tensor of the third column of systolic array. From top to bottom, they are: 00 K 00 +Q 01 K 01 +Q 02 K 02 、Q 00 K 10 +Q 01 K 11 +Q 02 K 12 、Q 10 K 10 +Q 11 K 11 +Q 12 K 12 、Q 10 K 00 +Q11 K 01 +Q 12 K 02 .

[0166] The multipliers 11 of the plurality of product-accumulators 10 of the fourth column of systolic array calculate the partial result tensors and input them to the adders 12. The adders 12 of the plurality of product-accumulators 10 of the fourth column of systolic array receive the partial result tensors of the fourth column of systolic array and the accumulated result tensors output by the third column of systolic array, and add them to obtain the accumulated result tensors of the fourth column of systolic array, which are sequentially from top to bottom as follows: 00 K 00 +Q 01 K 01 +Q 02 K 02 +Q 03 K 03 , Q 00 K 10 +Q 01 K 11 +Q 02 K 12 +Q 03 K 13 , Q 10 K 10 +Q 11 K 11 +Q 12 K 12 +Q 13 K 13 , Q 10 K 00 +Q 11 K 01 +Q 12 K 02 +Q 13 K 03 .

[0167] Similarly, when the plurality of product-accumulators 10 of the last column of systolic array obtain the accumulated result tensors of the fourth column of systolic array, the accumulated result tensors of the last column of systolic array are output through the registers 13 respectively and serve as the output result tensors of the corresponding rows. For example, in FIG. 12, the fourth column is the last column of systolic array, and after obtaining the accumulated result tensors of the fourth column of systolic array, the accumulated result tensors of the fourth column are output through the plurality of registers 13 of the fourth column. Among the accumulated result tensors of the fourth column, the accumulated result tensor of the first row is the output result tensor of the first row of systolic array, the accumulated result tensor of the second row is the output result tensor of the second row of systolic array, the accumulated result tensor of the third row is the output result tensor of the third row of systolic array, and the accumulated result tensor of the fourth row is the output result tensor of the fourth row of systolic array.

[0168] Referring to FIG. 2 and FIG. 3, the embodiment of the present application further provides a self-attention mechanism calculation array 100, the self-attention mechanism calculation array 100 comprises a plurality of product accumulators 10, the plurality of product accumulators 10 constitute a systolic array; the plurality of product accumulators 10 are used for calculating an intermediate tensor according to a feature tensor and a weight tensor; and the plurality of product accumulators 10 are further used for calculating a result tensor according to the intermediate tensor.

[0169] In some embodiments, each product accumulator 10 comprises a multiplier 11, a first selector 141 and a second selector 142, the first selector 141 and the second selector 142 are connected with the multiplier 11, the weight tensor comprises a first weight tensor, and the intermediate tensor comprises a first intermediate tensor; the first selector 141 is used for selecting the feature tensor to be input to the multiplier 11; the second selector 142 is used for selecting the first weight tensor to be input to the multiplier 11; and the multiplier 11 is used for multiplying the feature tensor and the first weight tensor to obtain the first intermediate tensor.

[0170] In some embodiments, each product accumulator 10 further comprises a register 13, the register 13 is connected with the multiplier 11; and the register 13 is used for storing the first intermediate tensor.

[0171] In some embodiments, the weight tensor further comprises a second weight tensor, and the intermediate tensor further comprises a second intermediate tensor; the second selector 142 is used for selecting the second weight tensor to be input to the multiplier 11; and the multiplier 11 is used for multiplying the feature tensor and the second weight tensor to obtain the second intermediate tensor.

[0172] In some embodiments, each product accumulator 10 further comprises a register 13, the register 13 is connected with the multiplier 11; and the register 13 is used for storing the second intermediate tensor.

[0173] In some embodiments, for the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the second selector 142 is used for selecting the first weight tensor to be input to the multiplier 11; for the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the multiplier 11 is used for multiplying the feature tensor and the first weight tensor to obtain the first intermediate tensor; wherein M is an odd number, and M≥1.

[0174] In some embodiments, for the plurality of product accumulators 10 of the Mth row of the systolic array, the second selector 142 is used for selecting the second weight tensor to be input to the multiplier 11;

[0175] For the plurality of product accumulators 10 of the Mth row of the systolic array, the multiplier 11 is used for multiplying the feature tensor and the second weight tensor to obtain the second intermediate tensor; wherein M is an odd number, and M≥1.

[0176] In some embodiments, the feature tensor comprises a plurality of channel sub-feature tensors, the weight tensor comprises a plurality of channel sub-weight tensors; the plurality of product accumulators 10 are configured to calculate a plurality of sub-intermediate tensors respectively at a plurality of time instants according to the plurality of channel sub-feature tensors and the plurality of channel sub-weight tensors; and the plurality of product accumulators 10 are configured to accumulate the plurality of sub-intermediate tensors to obtain the intermediate tensor.

[0177] In some embodiments, the register 13 is further connected to a second selector 142; the first selector 141 is configured to select the first intermediate tensor to be input to the multiplier 11; the second selector 142 is configured to select the second intermediate tensor from the register 13 to be input to the multiplier 11; the multiplier 11 is configured to multiply the first intermediate tensor and the second intermediate tensor to obtain a sub-result tensor; and the plurality of product accumulators 10 are configured to calculate the result tensor according to the corresponding plurality of sub-result tensors.

[0178] In some embodiments, the register 13 is further connected to a second selector 142; the first selector 141 is configured to select the second intermediate tensor to be input to the multiplier 11; the second selector 142 is configured to select the first intermediate tensor from the register 13 to be input to the multiplier 11; the multiplier 11 is configured to multiply the first intermediate tensor and the second intermediate tensor to obtain a sub-result tensor; and the plurality of product accumulators 10 are configured to calculate the result tensor according to the corresponding plurality of sub-result tensors.

[0179] In some embodiments, the first selector 141 of the Mth row of the systolic array is connected to the register 13 of the (M+1)th row of the systolic array;

[0180] For the plurality of product accumulators 10 of the Mth row of the systolic array, the first selector 141 is configured to select the first intermediate tensor from the (M+1)th row of the systolic array to be input to the multiplier 11; and for the plurality of product accumulators 10 of the Mth row of the systolic array, the second selector 142 is configured to select the second intermediate tensor from the register 13 to be input to the multiplier 11; wherein M is an odd number and M≥1.

[0181] In some embodiments, the first selector 141 of the (M+1)th row of the systolic array is connected to the register 13 of the (M+2)th row of the systolic array; for the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the first selector 141 is configured to select the second intermediate tensor from the (M+2)th row of the systolic array to be input to the multiplier 11; and for the plurality of product accumulators 10 of the (M+1)th row of the systolic array, the second selector 142 is configured to select the first intermediate tensor from the register 13 to be input to the multiplier 11; wherein M is an odd number and M≥1.

[0182] In some embodiments, the first selector 141 of the last row of the systolic array is connected with the register 13 of the first row of the systolic array; the first selector 141 is configured to select the second intermediate tensor from the first row of the systolic array as an input to the multiplier 11 for the plurality of product-accumulators 10 of the last row of the systolic array; the second selector 142 is configured to select the first intermediate tensor from the register 13 as an input to the multiplier 11 for the plurality of product-accumulators 10 of the last row of the systolic array; wherein M is an odd integer, and M≥1.

[0183] In some embodiments, the result tensor comprises a plurality of output result tensors; the plurality of product-accumulators 10 of each row of the systolic array is configured to accumulate a corresponding plurality of sub-result tensors to obtain an output result tensor of the row.

[0184] In some embodiments, each product-accumulator 10 further comprises a third selector 143 and an adder 12, the third selector 143, the multiplier 11 and the register 13 are all connected with the adder 12, the third selector 143 of the (N+1)th column of the systolic array is further connected with the register 13 of the Nth column of the systolic array; the third selector 143 is configured to select the accumulated result tensor from the Nth column of the systolic array as an input to the adder 12 for the plurality of product-accumulators 10 of the (N+1)th column of the systolic array; the multiplier 11 is configured to input the sub-result tensor of the (N+1)th column of the systolic array to the adder 12 for the plurality of product-accumulators 10 of the (N+1)th column of the systolic array; the adder 12 is configured to add the accumulated result tensor of the Nth column of the systolic array and the sub-result tensor of the (N+1)th column of the systolic array to obtain the accumulated result tensor of the (N+1)th column of the systolic array for the plurality of product-accumulators 10 of the (N+1)th column of the systolic array; the register 13 is configured to output the accumulated result tensor of the last column of the systolic array as an output result tensor of the row for the plurality of product-accumulators 10 of the last column of the systolic array; wherein N is an integer, and N≥2.

[0185] In some embodiments, the third selector 143 of the second column of the systolic array is further connected with the register 13 of the first column of the systolic array; the third selector 143 is configured to select the sub-result tensor from the first column of the systolic array as an input to the adder 12 for the plurality of product-accumulators 10 of the second column of the systolic array; the multiplier 11 is configured to input the sub-result tensor of the second column of the systolic array to the adder 12 for the plurality of product-accumulators 10 of the second column of the systolic array; the adder 12 is configured to add the sub-result tensor of the first column of the systolic array and the sub-result tensor of the second column of the systolic array to obtain the accumulated result tensor of the second column of the systolic array for the plurality of product-accumulators 10 of the second column of the systolic array; the register 13 is configured to output the accumulated result tensor of the second column of the systolic array for the plurality of product-accumulators 10 of the second column of the systolic array.

[0186] In some embodiments, for the plurality of product-accumulators 10 of the first column of systolic array, the multiplier 11 is configured to input the sub-result tensor of the first column of systolic array to the adder 12; and for the plurality of product-accumulators 10 of the first column of systolic array, the register 13 is configured to output the sub-result tensor of the first column of systolic array.

[0187] It should be noted that the above-mentioned implementation of the self-attention mechanism calculation method is also applicable to the self-attention mechanism calculation array 100 of the embodiments of the present application, and will not be described here.

[0188] Referring to FIG. 2 and FIG. 16, the embodiments of the present application further provide a self-attention mechanism calculation device 200, comprising the self-attention mechanism calculation array 100 of any of the above-mentioned embodiments.

[0189] Specifically, for a 4x4 self-attention mechanism calculation array 100, the matrix multiplication that can be completed in T cycles is "X 2×T ×W T×4 ", wherein "2xT" represents the size of the feature tensor, and "Tx4" represents the size of the weight tensor. For an MxN self-attention mechanism calculation device 200 (as shown in FIG. 16), the matrix multiplication that can be completed in T cycles of the self-attention mechanism calculation device 200 is "X 2M×T ×W T×4N " (as shown in FIG. 17).

[0190] Referring to FIG. 18, the embodiments of the present application further provide a self-attention mechanism calculation system 400. The self-attention mechanism calculation system 400 comprises one or more processors 410 and a memory 420, and the memory 420 stores a computer program. When the computer program is executed by the processor 410, the self-attention mechanism calculation method of any of the above-mentioned embodiments is implemented.

[0191] For example, when the computer program is executed by the processor 410, the self-attention mechanism calculation method is implemented as follows:

[0192] 010: calculating an intermediate tensor by the plurality of product-accumulators 10 according to the feature tensor and the weight tensor;

[0193] 020: calculating a result tensor by the plurality of product-accumulators 10 according to the intermediate tensor.

[0194] For another example, when the computer program is executed by the processor 410, the self-attention mechanism calculation method is implemented as follows:

[0195] 011: selecting the feature tensor input to the multiplier 11 by the first selector 141;

[0196] 012: selecting the first weight tensor by the second selector 142 to input to the multiplier 11;

[0197] 013: multiplying the feature tensor and the first weight tensor by the multiplier 11 to obtain a first intermediate tensor.

[0198] It should be noted that the foregoing embodiments of the self-attention mechanism calculation method and the self-attention mechanism calculation array 100 are also applicable to the self-attention mechanism calculation system 400 of the embodiments of the present application, and will not be described here.

[0199] Referring to FIG. 19, the embodiments of the present application also provide a computer readable storage medium 500, which stores a computer program 510. When the program is executed by a processor 520, the self-attention mechanism calculation method of any of the foregoing embodiments is implemented.

[0200] For example, when the program is executed by the processor 520, the self-attention mechanism calculation method as follows is implemented:

[0201] 010: calculating an intermediate tensor according to the feature tensor and the weight tensor by the plurality of multiply-accumulators 10;

[0202] 020: calculating a result tensor according to the intermediate tensor by the plurality of multiply-accumulators 10.

[0203] For another example, when the program is executed by the processor 520, the self-attention mechanism calculation method as follows is implemented:

[0204] 011: selecting the feature tensor by the first selector 141 to input to the multiplier 11;

[0205] 012: selecting the first weight tensor by the second selector 142 to input to the multiplier 11;

[0206] 013: multiplying the feature tensor and the first weight tensor by the multiplier 11 to obtain a first intermediate tensor.

[0207] It should be noted that the foregoing embodiments of the self-attention mechanism calculation method and the self-attention mechanism calculation array 200 are also applicable to the computer readable storage medium 500 of the embodiments of the present application, and will not be described here.

[0208] In summary, in the self-attention mechanism calculation method, the calculation array 100, the calculation device 200, the calculation system 400 and the computer readable storage medium 500 of the embodiments of the present application, the intermediate tensor is calculated according to the feature tensor and the weight tensor by the plurality of product accumulators 10, and then the result tensor is directly calculated based on the calculated intermediate tensor, without removing the intermediate tensor from the self-attention mechanism calculation array 100. In this way, the frequent access of the intermediate tensor between the self-attention mechanism calculation array and the external memory is avoided, the calculation efficiency is improved, and the problems of intermediate tensor reuse and storage optimization are solved.

[0209] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0210] Any process or method descriptions in flow charts or described elsewhere in this specification can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process. The flow diagrams and / or method descriptions can also be understood as representing some of the functions of a process that can be performed by a computer or a computer-aided system.

[0211] The logic and / or steps represented in the flow diagrams and / or described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a computer-readable storage medium can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable storage medium can specifically include the following: an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device; a portable computer diskette (magnetic); a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); a portable compact disc read-only memory (CDROM); and a paper tape or other punched tool. Additionally, the computer-readable storage medium can even be paper or other suitable medium upon which the program is printed, since the program can be electronically captured, for example, via the optical scanning of the paper or other medium, followed by the electronic conversion of the optically scanned program into a form that can be further processed by the computer. Data signals embodied in carrier waves, for example, are included in the computer-readable storage medium.

[0212] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0213] Those skilled in the art can understand that all or part of the steps of the method carried out by the above-mentioned embodiments can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof. In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically alone, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of software function module. The integrated module, if realized in the form of software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0214] Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A self-attention mechanism computing method, wherein, The application is applied to a self-attention mechanism calculation array (100), the self-attention mechanism calculation array (100) comprises a plurality of product accumulators (10), the plurality of product accumulators (10) constitute a systolic array, and the self-attention mechanism calculation method comprises: An intermediate tensor is calculated by the plurality of product accumulators (10) according to a feature tensor and a weight tensor; A result tensor is calculated by the plurality of product accumulators (10) according to the intermediate tensor.

2. The self-attention mechanism computing method of claim 1, wherein, Each product accumulator (10) comprises a multiplier (11), a first selector (141) and a second selector (142), the first selector (141) and the second selector (142) are connected with the multiplier (11), the weight tensor comprises a first weight tensor, the intermediate tensor comprises a first intermediate tensor, and the intermediate tensor is calculated by the plurality of product accumulators (10) according to a feature tensor and a weight tensor, comprising: The feature tensor is input into the multiplier (11) by the first selector (141); The first weight tensor is input into the multiplier (11) by the second selector (142); The feature tensor and the first weight tensor are multiplied by the multiplier (11) to obtain the first intermediate tensor.

3. The self-attention mechanism computation method of claim 2, wherein, Each product accumulator (10) further comprises a register (13), the register (13) is connected with the multiplier (11), and the self-attention mechanism calculation method further comprises: The first intermediate tensor is stored in the register (13).

4. The self-attention mechanism computing method of claim 3, wherein, The weight tensor further comprises a second weight tensor, the intermediate tensor further comprises a second intermediate tensor, and the intermediate tensor is calculated by the plurality of product accumulators (10) according to a feature tensor and a weight tensor, further comprising: The second weight tensor is input into the multiplier (11) by the second selector (142); The feature tensor and the second weight tensor are multiplied by the multiplier (11) to obtain the second intermediate tensor.

5. The self-attention mechanism computing method of claim 4, wherein, Each product accumulator (10) further comprises a register (13), the register (13) is connected with the multiplier (11), and the self-attention mechanism calculation method further comprises: The second intermediate tensor is stored in the register (13).

6. The self-attention mechanism computing method of any one of claims 2-5, wherein, The first weight tensor is input into the multiplier (11) by the second selector (142), comprising: For the plurality of product accumulators (10) in the (M+1)th row of the systolic array, the first weight tensor is input into the multiplier (11) by the second selector (142); The feature tensor and the first weight tensor are multiplied by the multiplier (11) to obtain the first intermediate tensor, comprising: For the plurality of product accumulators (10) in the (M+1)th row of the systolic array, the feature tensor and the first weight tensor are multiplied by the multiplier (11) to obtain the first intermediate tensor; Wherein, M is an odd number, and M is greater than or equal to 1.

7. The self-attention mechanism computing method of claim 4, wherein, The second weight tensor is selected by the second selector (142) to be input to the multiplier (11), including: For the plurality of product-accumulators (10) of the Mth row of the systolic array, the second weight tensor is selected by the second selector (142) to be input to the multiplier (11); The second intermediate tensor is obtained by multiplying the feature tensor and the second weight tensor by the multiplier (11), including: For the plurality of product-accumulators (10) of the Mth row of the systolic array, the second intermediate tensor is obtained by multiplying the feature tensor and the second weight tensor by the multiplier (11); Wherein, M is an odd number, M≥1.

8. The self-attention mechanism computing method of any one of claims 1-7, wherein, The feature tensor includes a plurality of channel sub-feature tensors, and the weight tensor includes a plurality of channel sub-weight tensors, and the intermediate tensor is calculated by the plurality of product-accumulators (10) according to the feature tensor and the weight tensor, including: A plurality of sub-intermediate tensors are calculated by the plurality of product-accumulators (10) according to the plurality of channel sub-feature tensors and the plurality of channel sub-weight tensors at a plurality of time points, respectively; The plurality of sub-intermediate tensors are accumulated by the plurality of product-accumulators (10) to obtain the intermediate tensor.

9. The self-attention mechanism computing method of claim 5, wherein, The register (13) is also connected with the second selector (142), and the result tensor is calculated by the plurality of product-accumulators (10) according to the intermediate tensor, including: The first intermediate tensor is selected by the first selector (141) to be input to the multiplier (11); The second intermediate tensor is selected by the second selector (142) from the register (13) to be input to the multiplier (11); The sub-result tensor is obtained by multiplying the first intermediate tensor and the second intermediate tensor by the multiplier (11); The result tensor is calculated according to a plurality of the sub-result tensors corresponding to the plurality of product-accumulators (10).

10. The self-attention mechanism computing method of claim 5, wherein, The register (13) is also connected with the second selector (142), and the result tensor is calculated by the plurality of product-accumulators (10) according to the intermediate tensor, including: The second intermediate tensor is selected by the first selector (141) to be input to the multiplier (11); The first intermediate tensor is selected by the second selector (142) from the register (13) to be input to the multiplier (11); The sub-result tensor is obtained by multiplying the first intermediate tensor and the second intermediate tensor by the multiplier (11); The result tensor is calculated according to a plurality of the sub-result tensors corresponding to the plurality of product-accumulators (10).

11. The self-attention mechanism computing method of claim 9, wherein, The first selector (141) of the Mth row of the systolic array is connected with the register (13) of the (M+1)th row of the systolic array, and the first intermediate tensor is selected by the first selector (141) to be input to the multiplier (11), including: For the plurality of product-accumulators (10) of the Mth row of the systolic array, the first intermediate tensor from the (M+1)th row of the systolic array is selected by the first selector (141) to be input to the multiplier (11); The selecting the first intermediate tensor from the register (13) to be input to the multiplier (11) by the second selector (142) comprises: For the plurality of product-accumulators (10) of the Mth row of the systolic array, the second intermediate tensor from the (M+1)th row of the systolic array is selected by the first selector (141) to be input to the multiplier (11); Wherein, M is an odd number, M≥1.

12. The self-attention mechanism computing method of claim 10, wherein, The first selector (141) of the (M+1)th row of the systolic array is connected with the register (13) of the (M+2)th row of the systolic array, and the selecting the second intermediate tensor to be input to the multiplier (11) by the first selector (141) comprises: For the plurality of product-accumulators (10) of the (M+1)th row of the systolic array, the first intermediate tensor from the register (13) is selected by the second selector (142) to be input to the multiplier (11); The selecting the first intermediate tensor from the register (13) to be input to the multiplier (11) by the second selector (142) comprises: For the plurality of product-accumulators (10) of the (M+1)th row of the systolic array, the first intermediate tensor from the register (13) is selected by the second selector (142) to be input to the multiplier (11); Wherein, M is an odd number, M≥1.

13. The self-attention mechanism computing method of claim 10, wherein, The first selector (141) of the last row of the systolic array is connected with the register (13) of the first row of the systolic array, and the selecting the second intermediate tensor to be input to the multiplier (11) by the first selector (141) comprises: For the plurality of product-accumulators (10) of the last row of the systolic array, the first intermediate tensor from the register (13) is selected by the second selector (142) to be input to the multiplier (11); The selecting the first intermediate tensor from the register (13) to be input to the multiplier (11) by the second selector (142) comprises: For the plurality of product-accumulators (10) of the last row of the systolic array, the first intermediate tensor from the register (13) is selected by the second selector (142) to be input to the multiplier (11); Wherein, M is an odd number, M≥1.

14. The self-attention mechanism computing method of claim 9 or 10, wherein, The result tensor comprises a plurality of output result tensors of rows, and the calculating the result tensor according to the plurality of sub-result tensors corresponding to the plurality of product-accumulators (10) comprises: The plurality of sub-result tensors corresponding to the plurality of product-accumulators (10) of each row of the systolic array are accumulated to obtain the output result tensor of the row.

15. The self-attention mechanism computing method of claim 14, wherein, Each of the product accumulators (10) further comprises a third selector (143) and an adder (12), the third selector (143), the multiplier (11) and the register (13) are connected with the adder (12), the third selector (143) of the (N+1)th column of the systolic array is further connected with the register (13) of the Nth column of the systolic array, and the accumulating of the plurality of sub-result tensors corresponding to the plurality of product accumulators (10) of each row of the systolic array to obtain the output result tensor of the row comprises: for the plurality of product accumulators (10) of the (N+1)th column of the systolic array, the third selector (143) is used to select the cumulative result tensor from the Nth column of the systolic array and input into the adder (12); for the plurality of product accumulators (10) of the (N+1)th column of the systolic array, the multiplier (11) is used to input the sub-result tensor of the (N+1)th column of the systolic array into the adder (12); for the plurality of product accumulators (10) of the (N+1)th column of the systolic array, the adder (12) is used to add the cumulative result tensor of the Nth column of the systolic array and the sub-result tensor of the (N+1)th column of the systolic array to obtain the cumulative result tensor of the (N+1)th column of the systolic array; for the plurality of product accumulators (10) of the last column of the systolic array, the register (13) is used to output the cumulative result tensor of the last column of the systolic array as the output result tensor of the row; wherein N is an integer, and N≥2.

16. The self-attention mechanism computing method of claim 15, wherein, The third selector (143) of the second column of the systolic array is further connected with the register (13) of the first column of the systolic array, and the accumulating of the plurality of sub-result tensors corresponding to the plurality of product accumulators (10) of each row of the systolic array to obtain the output result tensor of the row further comprises: for the plurality of product accumulators (10) of the second column of the systolic array, the third selector (143) is used to select the sub-result tensor from the first column of the systolic array and input into the adder (12); for the plurality of product accumulators (10) of the second column of the systolic array, the multiplier (11) is used to input the sub-result tensor of the second column of the systolic array into the adder (12); for the plurality of product accumulators (10) of the second column of the systolic array, the adder (12) is used to add the sub-result tensor of the first column of the systolic array and the sub-result tensor of the second column of the systolic array to obtain the cumulative result tensor of the second column of the systolic array; for the plurality of product accumulators (10) of the second column of the systolic array, the register (13) is used to output the cumulative result tensor of the second column of the systolic array.

17. The self-attention mechanism calculation method of claim 15 or 16, wherein, The accumulating of the plurality of sub-result tensors corresponding to the plurality of product accumulators (10) of each row of the systolic array to obtain the output result tensor of the row further comprises: For the plurality of product-sum accumulators (10) of the first column of the systolic array, the sub-result tensor of the first column of the systolic array is input to the adder (12) through the multiplier (11); For the plurality of product-sum accumulators (10) of the first column of the systolic array, the sub-result tensor of the first column of the systolic array is output through the register (13).

18. A self-attention mechanism computing array (100), wherein The self-attention mechanism calculation array (100) comprises a plurality of product-sum accumulators (10) constituting a systolic array; The plurality of product-sum accumulators (10) are configured to calculate an intermediate tensor according to a feature tensor and a weight tensor; The plurality of product-sum accumulators (10) are further configured to calculate a result tensor according to the intermediate tensor.

19. The self-attention mechanism computing array (100) of claim 18, wherein, Each product-sum accumulator (10) comprises a multiplier (11), a first selector (141) and a second selector (142), the first selector (141) and the second selector (142) are connected with the multiplier (11), the weight tensor comprises a first weight tensor, and the intermediate tensor comprises a first intermediate tensor; The first selector (141) is configured to select the feature tensor to be input to the multiplier (11); The second selector (142) is configured to select the first weight tensor to be input to the multiplier (11); The multiplier (11) is configured to multiply the feature tensor and the first weight tensor to obtain the first intermediate tensor.

20. The self-attention mechanism computing array (100) of claim 19, wherein, Each product-sum accumulator (10) further comprises a register (13) connected with the multiplier (11); The register (13) is configured to store the first intermediate tensor.

21. The self-attention mechanism computing array (100) of claim 20, wherein, The weight tensor further comprises a second weight tensor, and the intermediate tensor further comprises a second intermediate tensor; The second selector (142) is configured to select the second weight tensor to be input to the multiplier (11); The multiplier (11) is configured to multiply the feature tensor and the second weight tensor to obtain the second intermediate tensor.

22. The self-attention mechanism computing array (100) of claim 21, wherein, Each product-sum accumulator (10) further comprises a register (13) connected with the multiplier (11); The register (13) is configured to store the second intermediate tensor.

23. The self-attention mechanism computing array (100) according to any one of claims 19-22, wherein, For the plurality of product-sum accumulators (10) of the (M+1)th row of the systolic array, the second selector (142) is configured to select the first weight tensor to be input to the multiplier (11); For the plurality of product-sum accumulators (10) of the (M+1)th row of the systolic array, the multiplier (11) is configured to multiply the feature tensor and the first weight tensor to obtain the first intermediate tensor; Wherein, M is an odd number, and M is greater than or equal to 1.

24. The self-attention mechanism computing array (100) of claim 21, wherein, For the plurality of product-sum accumulators (10) of the Mth row of the systolic array, the second selector (142) is configured to select the second weight tensor to be input to the multiplier (11); For the plurality of product-sum accumulators (10) of the Mth row of the systolic array, the multiplier (11) is configured to multiply the feature tensor and the second weight tensor to obtain the second intermediate tensor; Wherein, M is an odd number, and M is greater than or equal to 1.

25. The self-attention mechanism computing array (100) according to any one of claims 18-25, wherein, The feature tensor comprises a plurality of channel sub-feature tensors, and the weight tensor comprises a plurality of channel sub-weight tensors; The plurality of product accumulators (10) are configured to calculate a plurality of sub-intermediate tensors respectively at a plurality of time points according to the plurality of channel sub-feature tensors and the plurality of channel sub-weight tensors; The plurality of product accumulators (10) are configured to accumulate the plurality of sub-intermediate tensors to obtain the intermediate tensor.

26. The self-attention mechanism computing array (100) of claim 22, wherein, The register (13) is further connected with the second selector (142); The first selector (141) is configured to select the first intermediate tensor to be input to the multiplier (11); The second selector (142) is configured to select the second intermediate tensor from the register (13) to be input to the multiplier (11); The multiplier (11) is configured to multiply the first intermediate tensor and the second intermediate tensor to obtain a sub-result tensor; The plurality of product accumulators (10) are configured to calculate the result tensor according to a plurality of the sub-result tensors.

27. The self-attention mechanism computing array (100) of claim 22, wherein, The register (13) is further connected with the second selector (142); The first selector (141) is configured to select the second intermediate tensor to be input to the multiplier (11); The second selector (142) is configured to select the first intermediate tensor from the register (13) to be input to the multiplier (11); The multiplier (11) is configured to multiply the first intermediate tensor and the second intermediate tensor to obtain a sub-result tensor; The plurality of product accumulators (10) are configured to calculate the result tensor according to a plurality of the sub-result tensors.

28. The self-attention mechanism computing array (100) of claim 26, wherein, The first selector (141) of the Mth row of the systolic array is connected with the register (13) of the (M+1)th row of the systolic array; For the plurality of product accumulators (10) of the Mth row of the systolic array, the first selector (141) is configured to select the first intermediate tensor from the (M+1)th row of the systolic array to be input to the multiplier (11); For the plurality of product accumulators (10) of the Mth row of the systolic array, the second selector (142) is configured to select the second intermediate tensor from the register (13) to be input to the multiplier (11); Wherein, M is an odd number, and M≥1.

29. The self-attention mechanism computing array (100) of claim 27, wherein, The first selector (141) of the (M+1)th row of the systolic array is connected with the register (13) of the (M+2)th row of the systolic array; For the plurality of product accumulators (10) of the (M+1)th row of the systolic array, the first selector (141) is configured to select the second intermediate tensor from the (M+2)th row of the systolic array to be input to the multiplier (11); For the plurality of product accumulators (10) of the (M+1)th row of the systolic array, the second selector (142) is configured to select the first intermediate tensor from the register (13) to be input to the multiplier (11); Wherein, M is an odd number, and M≥1.

30. The self-attention mechanism computing array (100) of claim 27, wherein, The first selector (141) of the last row of the systolic array is connected with the register (13) of the first row of the systolic array; For the plurality of product-accumulators (10) of the last row of the systolic array, the first selector (141) is configured to select the second intermediate tensor from the first row of the systolic array as an input to the multiplier (11); For the plurality of product-accumulators (10) of the last row of the systolic array, the second selector (142) is configured to select the first intermediate tensor from the register (13) as an input to the multiplier (11); wherein M is an odd integer, M≥1.

31. The self-attention mechanism computing array (100) according to claim 26 or 27, wherein The result tensor comprises a plurality of output result tensors; The plurality of product-accumulators (10) of each row of the systolic array are configured to accumulate the corresponding plurality of sub-result tensors to obtain the output result tensor of the row.

32. The self-attention mechanism computing array (100) of claim 31, wherein, Each of the product-accumulators (10) further comprises a third selector (143) and an adder (12), the third selector (143), the multiplier (11) and the register (13) are connected to the adder (12), the third selector (143) of the (N+1)th column of the systolic array is further connected to the register (13) of the Nth column of the systolic array; For the plurality of product-accumulators (10) of the (N+1)th column of the systolic array, the third selector (143) is configured to select the accumulated result tensor from the Nth column of the systolic array as an input to the adder (12); For the plurality of product-accumulators (10) of the (N+1)th column of the systolic array, the multiplier (11) is configured to input the sub-result tensor of the (N+1)th column of the systolic array to the adder (12); For the plurality of product-accumulators (10) of the (N+1)th column of the systolic array, the adder (12) is configured to add the accumulated result tensor of the Nth column of the systolic array and the sub-result tensor of the (N+1)th column of the systolic array to obtain the accumulated result tensor of the (N+1)th column of the systolic array; For the plurality of product-accumulators (10) of the last column of the systolic array, the register (13) is configured to output the accumulated result tensor of the last column of the systolic array as the output result tensor of the row; wherein N is an integer, N≥2.

33. The self-attention mechanism computing array (100) of claim 32, wherein, The third selector (143) of the second column of the systolic array is further connected to the register (13) of the first column of the systolic array; For the plurality of product-accumulators (10) of the second column of the systolic array, the third selector (143) is configured to select the sub-result tensor from the first column of the systolic array as an input to the adder (12); For the plurality of product-accumulators (10) of the second column of the systolic array, the multiplier (11) is configured to input the sub-result tensor of the second column of the systolic array to the adder (12); For the plurality of product-accumulators (10) of the second column of the systolic array, the adder (12) is configured to add the sub-result tensor of the first column of the systolic array and the sub-result tensor of the second column of the systolic array to obtain the accumulated result tensor of the second column of the systolic array; For the plurality of product-and-accumulate units (10) of the second column of systolic arrays, the register (13) is configured to output the accumulated result tensor of the second column of systolic arrays.

34. The self-attention mechanism computing array (100) according to claim 32 or 33, wherein For the plurality of product-and-accumulate units (10) of the first column of systolic arrays, the multiplier (11) is configured to input the sub-result tensor of the first column of systolic arrays to the adder (12). For the plurality of product-and-accumulate units (10) of the first column of systolic arrays, the register (13) is configured to output the sub-result tensor of the first column of systolic arrays.

35. A self-attention mechanism computing apparatus (200), wherein, The self-attention mechanism computing system (400) comprises one or more processors (410) and a memory (420), and the memory (420) stores a computer program, and the computer program is executed by the processor (410) to implement the self-attention mechanism computing method in any one of claims 1-17.

36. A self-attention mechanism computing system (400), wherein, The program is executed by the processor (520) to implement the self-attention mechanism computing method in any one of claims 1-17.

37. A computer readable storage medium (500) having stored thereon a computer program (510), wherein, The program is executed by the processor (520) to implement the self-attention mechanism computing method in any one of claims 1-17.

Images